The Theory Behind Contract Testing

What contract testing is and why it matters

Contract testing is a form of testing where you test that two systems have a shared understanding of expectations.

The contract is a JSON file containing the names of two systems interacting. It is the web interface interacting with an API. The contract lists all possible interactions between the two systems.

There is a shared broker (the contract) which the API can access directly. The API pulls down the contract from the shared broker and replays the expected request from the web app and then compares that all responses exist.

When there are two users of an API, for example a Mobile client and a Web App, a separate contract can be introduced to ensure that changes within the API won't break either of the applications.

Apart from detecting bugs missed by proper communication amongst teams, here are other reasons why contract testing can be helpful:

Speeds up development process and provides a faster feedback loop.
Enables teams to deploy changes independently and safely.
Issues are easier to debug since scope of tests with the contract is much smaller.
Developers can run contract tests and update the contract locally.
Promotes better code quality, removes the need to introduce unneeded requirements.
Ensures breaking changes will be communicated with teams and contract will always be up-to-date with these changes.

Contracts on the Consumer Side

Usually the contract is driven from the consumer of the API. Usually the consumer will receive business requirements for an application, thus they will create a contract for what fields that they will need.

The frontend engineer will

Define the interaction with a mock provider.
Runs the test against the mock provider and verifies the request is registered as an interaction.
If tests are successful, the contract will be generated automatically with includes the defined interaction. The frontend engineer will then upload the contract to a shared repository.

Contracts on the Provider Side

Contracts will generally be stored in a separate project repository, one of the backend engineers is notified if there are updates to the contract. The responsibility of the provider is to verify the contract.

The backend engineer will

Download the contract generated by the frontend engineer.
Verifies the expected contract will be returned from the API using a mock consumer.
If the actual response and the expected response are matching. The verification is successful.

Contract testing is only useful when you have both a provider side contract and a consumer side contract.

How Contract Testing Fits

Contract testing is in between the layer of Integration tests and Unit tests. They provide high confidence that API endpoints are shaped correctly and allow the developer to confidently know an endpoint will return the data needed. Providing early feedback to find any discrepancies and misunderstandings between two teams before deploying an application and running integration tests, mitigating compatibility issues found between E2E and integration tests.

Contract testing does not replace any of the aforementioned tests, it compliments them. See the diagram below when considering a common login scenario and where it should be covered by tests.

Contract testing does not solve:

Environment problems caused by configuration issues.
Networking issues such as timeouts and dropped requests.
Data integrity issues.
How the user interacts with an application.
Misunderstandings in business requirements.

When deciding whether to remove a integration or E2E test to a contract test, a good rule of thumb is to move tests that can be completely duplicated at the contract level. This gives you confidence that you don't need this as an integration or E2E test. Defining clear contracts between a consumer and a provider ensures that developers will only deliver functionalities that align with the consumers needs.

Technical Overview of Contract Testing

What is a Consumer?

Contract testing puts more power in the hands of the consumer. In day-to-day life, a consumer in a retail situation is a "user of a product or service". Each user may use their product slightly differently and/or have different uses of the product.

The consumer as it relates to us is generally a web or mobile application that makes requests to a backend service. But a consumer can also be a service that is calling another backend service.

In a consumer-driven approach to contract testing, the consumer is driving the contract. In the example of a login, a consumer dictates when the application requests the login data. An end user does not care about the shape or the data, they only care about whether they can log in or not. This is why the consumer is so integral and should influence the expected data response.

What is a Provider?

A provider is "something that someone needs or wants, if somebody wants your product, you provide them with it or make it available". In a consumer driven approach, the consumer will mock out the provider and the provider mocks out their downstream dependencies.

The provider must add a test pointing to the consumer within the contract testing process and will automatically get updates if they point to the latest version of the consumer contract.

What is a Contract?

A contract is “a legal document that states and explains a formal agreement between two different people or groups”. It is important to distinguish the differences between a contract and a schema. A schema is described as "an outline of a plain or theory". The schema provides the outline, it will be defined in the planning phases before development starts. A contract is used after development to ensure the schema does not change after released.

What is a Contract Broker?

A broker is “a person who talks to opposing sides, especially governments, making arrangements for them or bringing disagreements to an end“. A contract broker is the central place that contracts are stored. Without a broker, passing around contracts is manual and can lead to different versions of the most recent contract due to code changes.

Implementing Consumer Driven Contract Testing (CDCT)

The Focus of a Consumer Contract

A consumer contract is only supposed to make sure that when a consumer asks for data from a provider service, the correct data is returned as specified by the contract agreement. This does not care about the functional details of how the provider works, instead they focused on what the consumer needs and checks if they get it.

A general rule of thumb when deciding what to include in the contract is: "If I don't include this scenario, what bug in the consumer or what misunderstanding about how the provider responds might be missed. If the answer is none, don't include it." source

Building with LEGOs

It is important to understand coupling with contract testing. Think of coupling in the context of LEGO blocks. In a highly coupled scenario, LEGO blocks are intertwined and connected with other blocks, if you remove 1, the whole structure falls. These blocks are highly dependent on each other and depend on one block being present.

In contract testing you are able to smell a highly coupled contract if you make one small change to the provider and the consumer contract fails. If you're not careful, contract tests can easily become very brittle and flaky.

To combat flaky tests, Pact provides loose matchers, such as type based matchers. With type based matching, you care about what is the type of data returned, rather than the actual data itself. Consider the code sample below

const EXPECTED_BODY = {
  id: like(1),
  username: like("marie"),
  fullname: like("marie cruz"),
};

This object represents the shape the consumer expects from the data provider. If the consumer makes a GET request, it cares that the data exists, and the type of the data matches. Another question that Pact recommends is "If I made this looser/tighter, what bugs would I miss/prevent?"

Example Consumer Jest test

A consumer contract test can be broken down into 5 steps

Importing the required dependencies.
Setting up mock provider that the consumer will use.
Registering the expectations that the consumer will receive from the provider.
Verifying the consumer test and generate the contract.
Publish the contract to a broker.

const path = require("path");
const { fetchMovies } = require("./consumer");
const { PactV3, MatchersV3 } = require("@pact-foundation/pact");

const provider = new PactV3({
  dir: path.resolve(process.cwd(), "pacts"),
  consumer: "WebConsumer",
  provider: "MoviesAPI",
});

const EXPECTED_BODY = { id: 1, name: "My movie", year: 1999 };

describe("Movies API", () => {
  describe("When a GET request is made to /movies", () => {
    test("it should return all movies", async () => {
      provider
        .uponReceiving("a request for all movies")
        .withRequest({
          method: "GET",
          path: "/movies",
        })
        .willRespondWith({
          status: 200,
          body: MatchersV3.eachLike(EXPECTED_BODY),
        });

      await provider.executeTest(async (mockProvider) => {
        const movies = await fetchMovies(mockProvider.url);
        expect(movies[0]).toEqual(EXPECTED_BODY);
      });
    });
  });
});

The test above generates a contract WebConsumer-MoviesAPI.json using Pact to be uploaded to a shared broker.

Implementing CDCTs for Providers

When developing contract tests, it is essential to keep in mind that there are two sets of tests that you need to write, one for the consumer, the other for the providers. Provider contract tests typically demand less code than it's counterpart.

The Focus of a Provider Contract Test

The primary focus of a provider contract test is to verify the contract that the consumer has generated. Contract testing tools, such as Pact, provide a framework that allows data providers to pull the contract test and replay the interactions that the consumer registers as part of the contract.

A consumer-driven contract testing approach forces the provider to only develop features that the consumer requires. If the API introduces a breaking change that modify the type of a field, the contract should verify this change before applying it to production. To reiterate, contract testing should NOT verify validation, this should be a unit test.

If the provider changes the business logic of a validation, this would break a contract testing validation. Ideally business rules should not break a contract with a consumer.

Using Provider States Effectively

Provider states allow data providers to define the state a response needs to be in to be able to verify the interaction from the consumer contract successfully. When writing a provider contract test with provider states, you need to make sure that the provider state is provided from the consumer.

Set up the consumer test with a provider state.
Define the state of the provider.

Example Provider Jest Test

A provider contract test can be broken down into 5 steps

Importing the required dependencies.
Running the provider service.
Setting up the provider verifier options.
Writing the provider contract test.
Running the provider contract test.

const { Verifier } = require('@pact-foundation/pact');
const { importData, server } = require('./provider');

importData();

const port = '3001';
const app = server.listen(port, => console.log(`Listening on port ${port}...`));

const options = {
  provider: 'MoviesAPI',
  providerBaseUrl: `http://localhost:${port}`,
  pactBrokerUrl: process.env.PACT_BROKER_BASE_URL,
  pactBrokerToken: process.env.PACT_BROKER_TOKEN,
  providerVersion: '1.0.0',
  publishVerificationResult: true,
  consumerVersionTags: ['main'],
};

const verifier = new Verifier(options);

describe('Pact Verification', () => {
  test('should validate the expectations of movie-consumer', () => {
    return verifier
      .verifyProvider()
      .then(
        output => {
          console.log('Pact Verification Complete!');
          console.log('Result:', output);
          app.close();
      });
  });
});

Contract testing for GraphQL

An assumption is made that you are already familiar with the basics of GraphQL (writing queries, types, etc). If not, please pause and refer to https://graphql.org/learn/.

Let's assume that we have a GraphQL query to retrieve all movies in an example scenarios above. Following the same steps mentioned in Example Consumer Jest test.

Importing the Required Dependencies

Pact providers a lightweight wrapper GraphQLInteraction for GraphQL interactions within a contract test. Pact V3 currently does not support GraphQL interactions, so it is advised to use Pact V2 docs link PR link.