Contract testing at a glance

Author: Rotem Kirshenbaum

Testing microservices

Testing is a key ingredient in a successful CI/CD pipeline, especially in the microservices world. We test our classes in unit tests, test business processes in integration tests, even test our UI with visual tests. Our service is tested fully end to end — from frontend JavaScript code to our backend code and database.

However, services aren’t usually a single player in our deployment. Services are often interdependent, either directly by performing http requests to another service or indirectly by sending a message via a message bus.

How do I know that my newly deployed service won’t cause failures in services that depend on it, because I changed my API signature? How do I know that another service I depend on won’t cause my service to fail because now it’s sending a different message?

The simple solution is some form of end-to-end test. Simply deploy your service and its dependencies to a testing environment and run tests to verify that things are still working as expected across the environment.

This is fine when you have a handful of microservices. What happens when you have dozens? Hundreds? Thousands?

This is fine

The dependency chain between services can grow and become an expensive, time consuming and laborious undertaking — I just want to test my service before deployment.

Enter Contract Testing

Contract testing is a testing paradigm that aims to solve this issue. Instead of running expensive end-to-end tests, we define a contract between 2 entities: the consumer and the provider.

A contract describes the interaction between both sides, as a set of requirements that the consumer has from the provider. It’s analogous to interfaces in OOP — The interface is the contract between the calling code (consumer) and the implementing class (provider).

For example, the consumer may declare that when it performs a POST request to a certain endpoint, and that it will receive a response with an ‘OK’ HTTP status and a JSON body:

"consumer": {
    "name": "my-consumer"
  "provider": {
    "name": "my-provider"
  "interactions": [
      "_id": "fd0c3e907b1e128d241810303938b04884b3f242",
      "description": "Get data from provider",
      "request": {
        "method": "POST",
        "path": "/some/path"
      "response": {
        "status": 200,
        "headers": {
          "Content-Type": "application/json"
        "body": true

The contract can also define that this JSON will contain the fields we require in the correct format and type.

Once we have a contract, both consumer and provider can verify themselves:

  1. The consumer will receive a mocked response based on the contract.
  2. The provider will verify that it can generate the expected response according to a given request.

The important concept to take away from this is that both of these tests occur independently of each other. We test the consumer and provider separately, whenever we need and want, without the need to deploy both of them to a test environment.

This is the power of contract testing!

So how do we actually do it? For that there’s Pact.

What is PACT ?

Baby don’t hurt me, no more

Pact is a contract testing tool; it defines a format for describing a contract (a JSON file) and an API.

On the consumer side, the Pact API will generate a contract based on the consumer requirements from a specific provider and set up a mock http server that returns a result accordingly.

The consumer can then call this HTTP endpoint and verify that it can actually handle the response correctly.

For the provider side, we load the contract file and let Pact perform the HTTP call to the provider service. Pact then verifies that the provider actually returned the correct response based on the contract.

How does the consumer and the provider share the contract between them? 
These are 2 different services that can even reside on different code repos.

For that Pact has another tool in its belt, which is called the Pact Broker. Essentially, this is a repository of contracts; when a consumer generates a contract it publishes it to the broker. A provider then can ask the broker for relevant contracts, using a rest API.

Once a contract succeeded or failed verification, we update it in the pact broker.

The broker serves a very important task — it’s the leading authority on whether or not we can actually deploy a service to an environment. Only the broker knows which versions of services work well with each other.

Pact at Tipalti

So how are we using Pact at Tipalti? How does Pact affect our CI/CD pipeline?

First, there are some things to consider:

  1. Versioning — each consumer / provider in a contract verification is versioned by the git commit id of the relevant code.
  2. Tags — each consumer / provider in a contract verification is tagged with the relevant branch name and any environment it was deployed to (QA, sandbox, production).

This information both allows us to understand the participants of the contract verification and allows us to query the Pact broker.

Let’s take a look at the flow:

Contract tests

The contract tests differ between consumers and providers:

Consumers will verify that they can handle the provider’s response according to that contract.

Providers will query the Pact broker for the relevant consumers and check that the provider responds correctly to each one of their contracts; we search for contracts that are tagged as “prod”, since we want to be sure that we won’t break our production environment with our changes.

These are implemented as unit tests, based on a set of test suites we prepared as part of our testing framework.

Local development / Pull request

Contract tests will run as part of local development and PR tests, the same as any other tests. The results of the contract verification is not published to the Pact broker since the code is not yet part of one of our main development branches.

Post PR

After a PR is completed, we trigger another run of the contract tests on the target branch. 
This time the contract test results are published to the Pact broker and tagged accordingly.

Consumers also trigger running contract tests for all relevant providers that are tagged with “prod”. This verifies the contract on both ends (consumer and provider).

Since these builds run in the background, after the consumer PR, we notify the build results via Slack to the relevant team.

Release pipeline

The first step of the release pipeline is to check if we can deploy the current commit. This is done via the aptly named Pact broker “can-i-deploy” command line.

We run this tool to verify that the commit id (which is also the version of the consumer / provider) can be deployed to the “prod” environment:

pact-broker can-i-deploy --pacticipant "MyServiceName" --version 23jsa45bg --to "prod" additional parameters omitted

If this step succeeds, we can continue in our release pipeline and deploy our service.
On each deployment to an environment we also tag the contract with the name of the environment.

If this step fails, this means that our service has an invalid contract and we can’t continue in our release pipeline — thus achieving our goal of protecting our production environment from failing due to our service.


Contract testing is an important tool in our testing arsenal that allows us to easily verify the interactions between our microservices. We use Pact and Pact broker as our contract testing tools to generate, verify and publish our contract tests.

NodeJS MS-SQL integration testing with Docker/Mocha

Author: Zachary Leighton

Integration testing vs. unit testing

Unit tests are great for testing a function for a specific behavior, if you code right and create your “mocked” dependencies right you can be reasonably assured of your code’s behavior.

But our code doesn’t live in isolation, and we need to make sure all the “parts” are connected and working together in the way we expect. This is where integration tests come in to play.

Charlie Sheen didn’t write unit tests, and look where that got him

A good way to explain the difference would be that a unit test would test that a value (let us say an email for simplicity) passes a test for business logic (possibly a regex or something — maybe checks the URL) and the email and rules would be provided as mocks/stubs/hard-coded in the test, while an integration test of this would check the same logic but also retrieve the rules and value from a database — thus checking all the pieces fit together and work.

If you want more examples or want to read up a bit more on this there are great resources on Medium, as well as Stack Overflow, etc. The rest of this article will assume you are familiar with NodeJS and testing it (here we use Mocha — but feel free to use whatever you like).

Pulling the MS-SQL image

He used Linux containers 🙂

To start you’ll want to pull the Docker image, simply run the command docker pull microsoft/mssql-server-linux:2017-latest (Also if you haven’t installed Docker you might want to do that too 😃)

This might take a few minutes depending on what you have installed in your Docker cache.

After this is done, please make sure to right click, go to “Settings…” and enable: “Expose daemon on tcp://localhost:2375”. As we will see in a few sections this needs to be set to process.env.DOCKER_HOST for the Docker modem to run correctly.

Delaying Mocha for setup

Since we need a few moments to spin up the container and deploy the schema we will use the --delay flag for Mocha.

This adds a global function run() that needs to be called when the setup is done.

You should also use the --exit flag which will kill Mocha after the test run, even if a socket is open.

Preparing the run

In this example, we use the --require flag to require a file before the test run. In this file an IIFE (immediately invoked function expression) is used because we need to call some async functions and await them, and then call the done() function from above. This can be done with callbacks but it is not so clean.

The IIFE should end up looking like this:

(async () => {
    const container = require('./infra/container');
    await container.createAsync();
    await container.initializeDbAsync();
    run(); // this kicks off Mocha
    beforeEach(async () => {
        console.log('Clearing db!');
        await container.clearDatabaseAsync();
    after(async () => {
        console.log('Deleting container!');
        await container.deleteAsync();

Spinning up the container from Node

In the above IIFE we have the method container.createAsync(); which is responsible for setting up the container.

const { Docker } = require('node-docker-api');
const docker = new Docker();
async function createAsync() {
    const container = await docker.container.create({
        Image: 'microsoft/mssql-server-linux:2017-latest',
        name: 'mssqltest',
        ExposedPorts: { '1433/tcp': {} },
        HostConfig: {
            PortBindings: {
                '1433/tcp': [{ HostPort: '<EXPOSED_PORT>' }]
        Env: ['SA_PASSWORD=<S00p3rS3cUr3>', 'ACCEPT_EULA=Y']
    console.log('Container built.. starting..');
    await container.start();
    console.log('Container started... waiting for boot...');
    sqlContainer = container;
    await checkSqlBootedAsync();
    console.log('Container booted!');

The container is created from the async method docker.container.create , the docker instance needs to have process.env.DOCKER_HOST set, in our case we have a local Docker server running (see: Pulling the MS-SQL image) so we’ll use that.

The options come from the modem dockerode and it uses the Docker API.

After the container spins up we need to check that SQL finished running, our port is <EXPOSED_PORT> and the password is <S00p3rS3cUr3> (these are placeholders so make sure you put something valid).

If you want to read more about what is happening here with the EULA option, etc. check out the guide here from Microsoft.

Since it takes a few seconds for the SQL server to boot up we want to make sure it is running before firing off the test suite. A solution we came up with here was to continually try and connect for 15 seconds every 1/2 second and when it connects, exit.

If it fails to connect within 15 seconds something went wrong and we should investigate further. The masterDb.config options should line up with where you’re hosting Docker and on what port you’re exposing 1433 to the host. Also remember the password you set for sa .

async function checkSqlBootedAsync() {
    const timeout = setTimeout(async () => {
        console.log('Was not able to connect to SQL container in 15000 ms. Exiting..');
        await deleteAndExitAsync();
    }, 15000);
    let connecting = true;
    const mssql = require('mssql');
    console.log('Attempting connection... ');
    while (connecting) {
        try {
// don't use await! It doesn't play nice with the loop 
            mssql.connect(masterDb.config).then(() => {
                connecting = false;
        catch (e) {
            // sink
        await sleep(500);

Deploying db schema using Sequelize

Fun Fact: Liam Neeson used Docker to release the Kraken as well.

We can quickly use Sequelize to deploy the schema by using the sync function, then as we will see below it is recommended to set some sort of flag to prevent wiping of a non-test DB.

First though, we want to actually create the db using the master connection. The code will end up looking something like this:

async function initializeDbAsync() {
    const sql = 'CREATE DATABASE [MySuperIntegrationTestDB];';
    await masterDb.queryAsync(sql, {});
    await sequelize.sync();
    return setTestingDbAsync();

Safety checks

Let’s face it, if you’ve been programming professionally for any reasonable amount of time — you’ve probably dropped a database or file system.

And if you haven’t go run out and buy a lotto ticket because man you’re lucky.

This is the reason to set up infrastructure for backups and other things of the sort, roadblocks if you will, to prevent human error. While this integration test infrastructure you just finished setting up here is great, there is a chance you may have misconfigured the environment variables, etcetera.

I will propose here one possible solution, but feel free to use your own (or suggest more in the comments!).

Here we will use the SystemConfiguration table and have a key value pair on key TestDB that’s value needs to be truthy for the tables to be truncated. Also at multiple steps I recommend checking the NODE_ENV environment variable to be test which can make sure you didn’t accidentally run this code in a non-test environment.

At the end of the last section we saw the call to setTestingDbAsync the content is as follows:

async function setTestingDbAsync() {
    const configSql =
        "INSERT INTO [SystemConfiguration] ([key], [value]) VALUES (?, '1')";
    return sequelize.query(configSql, {replacements: [systemConfigurations.TestDB]});

This sets the value in the database, which we will check for in the next snippit. Here is a snippet of code that will check the existence of a value on the key TestDB (provided from a consts file) that we just set.

const result = await SystemConfiguration.findOne({ where: {key: systemConfigurations.TestDB }});
    if (!result) {
        console.log('Not test environment, missing config key!!!!');
        // bail out and clean up here
// otherwise continue

Wiping the test before each run

Taking the code above and combining it with something to clear the database we come up with the following function:

const useSql = 'USE [MySuperIntegrationTestDB];';

async function clearDatabaseAsync() {
    const result = await SystemConfiguration.findOne({ where: {key: systemConfigurations.TestDB }});
    if (!result || !result.value) {
        console.log('Not test environment, missing config key!!!!');
        await deleteAndExitAsync();
    const clearSql = `${useSql}
       EXEC sp_MSForEachTable 'DISABLE TRIGGER ALL ON ?'
       EXEC sp_MSForEachTable 'DELETE FROM ?'
       EXEC sp_MSForEachTable 'ENABLE TRIGGER ALL ON ?'`;
    await sequelize.query(clearSql);
    return setTestingDbAsync();
async function setTestingDbAsync() {
    const configSql = "INSERT INTO [SystemConfiguration] ([key], [value]) VALUES (?, '1')";
    return sequelize.query(configSql, {replacements: [systemConfigurations.TestDB]});

This will check for the existence of the value for key TestDB in the SystemConfiguration table before continuing. If it isn’t there it will exit the process.

Now how does this run within the context of Mocha?

If you remember in the IIFE we had a call to beforeEach , this is where you want to have this hook so that you have a clean database for each test.

beforeEach(async () => {
        console.log('Clearing db!');
        await container.clearDatabaseAsync();

Shutdown / Teardown

You don’t want to leave the Docker in an unknown state, so at the end of the run simply kill the container, you’ll want to use force too.

Docker reached out to us and said they don’t use exhaust ports

The after look looks like this:

after(async () => {
        console.log('Deleting container!');
        await container.deleteAsync();

And the code inside container.deleteAsync(); looks like this:

async function deleteAsync() {
    return sqlContainer.delete({ force: true });

Putting it all together

Since this article was a bit wordy, and jumped around a bit here are the highlights of what to do to get this to work:

  • Delay Mocha using --delay
  • Require a setup script and use an IIFE to set up the container/DB
  • Spin up a Docker container instance, wait for SQL to boot
  • Deploy the schema using Sequelize and also put in a safety check so we don’t wipe a non-test DB.
  • Hook the wipe logic into the beforeEach hook
  • Hook the teardown logic into the after hook
  • Create amazing codez and test them

I hope you enjoyed this article, and suggestions, comments, corrections and more memes are always welcomed.

Good luck and happy testing!