GOV.UK Pay uses a microservices architecture. We used to run end-to-end tests to verify the whole system worked. This meant spinning up all our microservices in a DockerCompose environment to run tests which was:
- time-consuming
- unsustainable
- resource intensive
- not giving us the benefits of our microservice architecture as we were testing as a monolith
Contract testing is a methodology which tests if applications will work together without deploying them. By switching to contract testing we now have confidence that any service we deploy will always be compatible with the things it needs to connect with.
Contract testing in our build pipeline
Using contract testing gave us lots of benefits but also introduced some complexity into our build pipeline. These are the relevant components in the pipeline:
- Consumer - this is the service consuming an API from another service known as the “provider”. Contracts are consumer-driven - the consumer defines the contract or pact.
- Provider - the service providing an API.
- Pact Broker - the web application storing the contracts. It keeps track of which versions of consumers are compatible with corresponding versions of providers.
- A continuous integration (CI) tool.
Consumer CI Build Pipeline
We will not explain the Build, Test and Deploy stages as these should be familiar to technical readers. Once a developer makes a commit to GitHub, an automated script runs the following actions.
Publish Pact
If the build passes, the pact(s) are published to the pact broker along with its version (its Git commit SHA) and tagged with the branch name, for example, master.
Run pact provider test(s)
A consumer may have contracts between one or more providers. Here’s an example Jenkins configuration where the build script runs the provider tests twice:
ws('contract-tests-wp') { runPactProviderTests("pay-direct-debit-connector", "${env.PACT_TAG}") runPactProviderTests("pay-connector", "${env.PACT_TAG}") }
The runPactProviderTests job checks out the master branch of a provider project and runs the pact(s) published by the consumer.
Why run a provider build here and not use a Webhook to kick-off a provider build? We decided we wanted instant feedback on the provider build, rather than having a provider master build break without clarity on which consumer build caused it.
Run end-to-end tests
Moving existing end-to-end tests to contract testing will take some time so we still have to run these.
Check pact compatibility
The pact broker is asked if this version of the consumer is compatible with the version of the provider currently in the test environment. You can do this by using the pact broker cli or ruby client. If this check passes, the service is deployed.
Tag consumer version
Tell the pact broker this version of the consumer is the one currently in the test environment by tagging the version with test.
Provider CI Build Pipeline
Run provider contract test and publish verification result
In this step, the provider contract test(s) retrieves the relevant consumer pact(s) tagged with master from the pact broker and runs the pact(s) against itself.
If the test(s) pass, a verification result of ‘true’ is sent to the pact broker. This tells the pact broker that this version of the provider is compatible with the version of consumer in the master branch.
Check pact compatibility
Check whether the provider build is safe to deploy against the version of the consumer(s) currently in the test environment.
Staging and Production Deployment Pipeline
Here we check the version of the consumer or provider we want to deploy is compatible with the version of the provider or consumer currently in the staging or production environment.
Issues with adding contract testing
Introducing contract testing meant we had to deal with various issues. Our most complex problem was dealing with circular dependencies.
For a while, contract changes at the consumer side were breaking changes to the provider. This meant we would have to make changes in the provider and trigger a provider build before we could run a consumer build successfully. But, what happens if a contract is updated (for example, to add a new interaction), without requiring the provider to change its API?
Consider this scenario:
The contract is updated by a developer (C2) but without requiring the provider to change its API:
The consumer build runs well up until the deploy to the test environment. At this point, because the provider build has not verified C2, the compatibility check fails. There is no provider tagged with test that has verified C2. Or, to put it another way, the current provider in the test environment has not verified C2.
We can not deploy the consumer to the test environment, because there’s no provider tagged with test for the C2. We can kick off a provider build which will verify C2 (with tag master) but it will fail to deploy as there is no consumer tagged with test for the C2 pact.
This is a circular dependency.
Remember the provider pact tests gets the relevant consumer pacts tagged with master from the pact broker and runs the pacts against itself. The solution is to run the provider test with the consumer pacts tagged with master, test, staging, and production:
@PactBroker(host = "pact-broker-host", tags = {"master", "test", "staging", "production"})
If the provider test passes, it means any pact in our environments is compatible with the provider that is about to be deployed. The downside now is that you now have to run the provider test 4 times but this ensures backward compatibility. We did not have a problem with this because contracts rarely change.
With this solution in place, when we re-run the scenario as described above the compatibility check still fails. We then carry out the following steps.
1. Trigger another provider build
The pact broker will now know the provider version is compatible with C2 as well as C1, which is the version currently in the test environment. The provider will then deploy ok.
2. Re-trigger the consumer build
The compatibility check will now pass and the script deploys the consumer.
Our build pipeline also introduces tight coupling of contracts between consumers and providers. Sometimes, a developer may want to commit a consumer contract into the branch and implement the provider later. To get around this, we do not publish the pact(s) to the pact broker so it has no knowledge of it. One way of doing it is to exclude those pacts in the build file. We do this using some custom code.
Benefits of contract testing
Using contract testing has allowed us to:
- only use end-to-end tests for new functionality when it is unavoidable
- be confident deployments will not introduce breaking changes between services
- be in a position to replace expensive end-to-end testing with pact testing where applicable
Our biggest tip if you’re thinking about using contract testing is to introduce each bit of functionality into your build pipeline incrementally. You should also have learning sessions to educate your team on how contract testing works. We found that whenever we get errors, it can take a good deal of logical thinking to solve the problem. Some common errors are:
- circular dependencies, which we talked about
- a pact did not get published or verified for some reason, leading to a compatibility check problem when deploying
The Pact Nirvana documentation is also useful and you can join the Pact Foundation Slack channel.
Have you successfully used contract testing? If you have any questions, leave a comment below.
