[WIP]: jest watch placeholder

[YOU54F]

Placeholder for integration with jest watch mode

Considerations

• should run mock service on random ports to avoid clashes
• should only run changed tests when In watch mode
• we need to exclude the log and pact directories to avoid getting stuck in infinite loops
• write mode is important in the pact options. We need to ensure the following
  • generated pacts should not contain old interactions
  • generated pacts must contain all interactions even if we test the same consumer/provider pair across multiple test files
  • deleting all pacts before a single test run solves that problem but if running in watch mode we can’t delete all pacts, as jest-pact May only run a single changed pact and deleting them all would mean we would be missing pact files ( overwrite seems like the best option when in watch mode, but we need to be careful to ensure all pact interactions are recorded.
  • we can get the name of a test that has failed interaction, and jest watch will rerun a full test file, not just a single test within the file, so we could just delete the associated pact file but need to consider if the user has two tests with the same consumer/provider pair. We don’t want to delete the pact and only rerun a single test as we will lose interactions in our pact contract

https://jestjs.io/docs/en/watch-plugins

Existing watch plugins

• jest-watch-directories Select directories to test.
• jest-watch-exec Execute scripts during the watch cycle.
• jest-watch-lerna-packages Select Lerna packages to test.
• jest-watch-master Check changes since master.
• jest-watch-repeat Repeat test runs multiple times.
• jest-watch-select-projects Select which Jest projects to run.
• jest-watch-suspend Suspend watch mode so that your changes would not trigger test runs.
• jest-watch-toggle-config Toggle boolean settings (e.g. verbosity, test coverage).
• jest-watch-typeahead Filter your tests by file name or test name.
• jest-watch-yarn-workspaces Select Yarn workspaces to test.
• node-recorder Toggle recording modes for node-recorder.

[YOU54F]

looks like adding

watchPathIgnorePatterns: ["pact/logs/*","pact/pacts/*"] into a consuming projects jest.config.js lets watch mode run without getting into an infinite loop.

As we set the log & pact output directories, we should be able to hook into the API and ignore them (and also support user configured paths)

module.exports = {
  globals: {
    "ts-jest": {
      tsConfig: "tsconfig.json"
    }
  },
  moduleFileExtensions: ["ts", "js", "json"],
  transform: {
    "^.+\\.(ts|tsx)$": "ts-jest"
  },
  testMatch: ["**/*.test.(ts|js)", "**/*.it.(ts|js)", "**/*.pacttest.(ts|js)"],
  testEnvironment: "node",
  reporters: ["default", "jest-junit"],
  watchPathIgnorePatterns: ["pact/logs/*","pact/pacts/*"]
};

[YOU54F]

re-running only failed tests - https://github.com/pact-foundation/pact-js#re-run-specific-verification-failures

Re-run specific verification failures

If you prefix your test command (e.g. npm t) with the following two environment variables, you can selectively run a specific interaction during provider verification.

For the e2e example, let's assume we have the following failure:

3 interactions, 2 failures

Failed interactions:

* A request for all animals given Has some animals

* A request for an animal with id 1 given Has an animal with ID 1

If we wanted to target the second failure, we can extract the description and state as the bits before and after the word "given":

PACT_DESCRIPTION="a request for an animal with ID 1" PACT_PROVIDER_STATE="Has an animal with ID 1" npm t

Also note that PACT_DESCRIPTION is the failing description and PACT_PROVIDER_STATE is the corresponding providerState from the pact file itself.

[YOU54F]

jest notes https://github.com/pact-foundation/pact-js#timeout

Note on Jest

Jest uses JSDOM under the hood which may cause issues with libraries making HTTP request.

You'll need to add the following snippet to your package.json to ensure it uses
the proper Node environment:

"jest": {
  "testEnvironment": "node"
}

Also, from Jest 20, you can add the environment to the top of the test file as a comment. This will allow your pact test to run along side the rest of your JSDOM env tests.

/**
 * @jest-environment node
 */

Jest also runs tests in parallel by default, which can be problematic with Pact which is stateful. See parallel tests to see how to make it run in parallel, or run Jest with the --runInBand option to run them sequentially.

See this issue for background,
and the Jest example for a working example.

[YOU54F]

parallel execution - what is the best pactfile write mode. How can we avoid

• leaving old interations in pacts
• ensuring every pact from a parallel run is in the pact file
• need to consider if we have a consumer/provider test across multiple files, (merge mode)
• as we know where the pacts are stored, should be just delete them at the start of every run?

Parallel tests https://github.com/pact-foundation/pact-js#parallel-tests

Pact tests are inherently stateful, as we need to keep track of the interactions on a per-test basis, to ensure each contract is validated in isolation from others. However, in larger test suites, this can result in slower test execution.

Modern testing frameworks like Ava and Jest support parallel execution out-of-the-box, which

The good news is, parallel test execution is possible, you need to ensure that:

1. Before any test run invocation, you remove any existing pact files, to prevent invalid / stale interactions being left over from previous test runs
2. Each test is fully self-contained, with its own mock server on its own port
3. You set the option pactfileWriteMode to "merge", instructing Pact to merge any pact documents with the same consumer and provider pairing at the end of all test runs.

When all of your tests have completed, the result is the union of the all of the interactions from each test case in the generated pact file.

See the following examples for working parallel tests:

• Pact with AVA (Node env)
• Pact with Mocha

Splitting tests across multiple files

Pact tests tend to be quite long, due to the need to be specific about request/response payloads. Often times it is nicer to be able to split your tests across multiple files for manageability.

You have a number of options to achieve this feat:

1. Consider implementing the Parallel tests guidelines.

2. Create a Pact test helper to orchestrate the setup and teardown of the mock service for multiple tests.

   In larger test bases, this can significantly reduce test suite time and the amount of code you have to manage.

   See this example and this issue for more.

3. Set pactfileWriteMode to merge in the Pact() constructor

   This will allow you to have multiple independent tests for a given Consumer-Provider pair, without it clobbering previous interactions, thereby allowing you to incrementally build up or modify your pact files.

   This feature addresses the use case of "my pact suite takes bloody ages to run, so I just want to replace the interactions that have been run in this test execution" and requires careful management

   NOTE: If using this approach, you must be careful to clear out existing pact files (e.g. rm ./pacts/*.json) before you run tests to ensure you don't have left over requests that are no longer relevent.

   See this PR for background.

[TimothyJones]

Lots to unpack here, thanks for writing this all up.

If possible, I'd like not to bake in assumptions about how people have set up their environment. For example, not everyone will have their tests output to <project-root>/pact/pacts/ (mine are just <project-root>/pacts). Same for the log files.

Similarly, I name my tests .pact.test.js, so that they're picked up without extra configuration - but perhaps this isn't what everyone wants to do (I think some of the examples had pact and the rest of the unit tests separated).

Ideally we'd read all of this from config somehow, but it's not clear where the best place is (there's a little bit of discussion here). I think probably package.json, but it seems likely that some of that would want to be picked up by pact-js directly.

I think we'd want to configure the jest plugin to clean out the pacts directory on start, then use update for the pact file write mode. This avoids keeping removed interactions around, while still allowing multiple test files to describe the same consumer/provider pairing.

I think the best value is "what would the expected functionality be" - how can we help developers not have to think about the way pact is configured, and have it work for the majority of setups?

[YOU54F]

Lots to unpack here, thanks for writing this all up.

No worries homie, yeah quite a few things to consider.

If possible, I'd like not to bake in assumptions about how people have set up their environment. For example, not everyone will have their tests output to <project-root>/pact/pacts/ (mine are just <project-root>/pacts). Same for the log files.

Agree with you completely here about not wanting to bake in assumptions. The less opinionated we can build the adaptor the better. We can take a user defined location by exposing dir and log options. (we should probably allow all configuration options that pact-js ideally by using their type definition).

export interface PactOptions {
  provider: string;
  consumer: string;
  port?: number;
  logLevel?: LogLevel;
  pactfileWriteMode?: PactFileWriteMode;
  dir?: dir,
  log?: log
}

By taking in user provided paths, we can then support it on our defaults

const applyDefaults = (options: PactOptions) => ({
  port: options.port,
  log: path.resolve(
    process.cwd(),
    `${dir}`,
    `${options.consumer}-${options.provider}-mockserver-integration.log` || path.resolve(
    process.cwd(),
    "logs",
    `${options.consumer}-${options.provider}-mockserver-integration.log`
  ),
  dir: path.resolve(process.cwd(), `${log}`) || path.resolve(process.cwd(), "pacts"),
  spec: 2,
  logLevel: options.logLevel || "error",
  pactfileWriteMode: options.pactfileWriteMode || "update",
  ...options
});

Just thinking about that further, that would mean that a user would need to provide the log/dir directories every time they use pactWith() which is less than ideal, so your latter point about reading from config sounds much slicker. That is pretty easily done, I've written something in another project where I need to read version numbers from my package.json for the CLI tool to display it.

let version;
try {
  const json = JSON.parse(
    fs.readFileSync(
      "./node_modules/cypress-slack-reporter/package.json",
      "utf8"
    )
  );
  version = json.version;
} 
 catch (e) {
    version = "Cannot determine version";
  }

Similarly, I name my tests .pact.test.js, so that they're picked up without extra configuration - but perhaps this isn't what everyone wants to do (I think some of the examples had pact and the rest of the unit tests separated).

The adaptor doesn't need to know what the files are called, that it up to the end-user and their own configuration setup. The only thing we need to know about really is where the generated pact contracts & logs live, so we can get rid of them on each test run, and ignore them with a watchPathIgnorePatterns: ["<pactDir>/*","<logDir>/*"]. I will have to see if this is something the adaptor can do, or if the end-user will just need to add this into their own jest config (be it in package.json, jest.*.js config file or wherever

Ideally we'd read all of this from config somehow, but it's not clear where the best place is (there's a little bit of discussion here). I think probably package.json, but it seems likely that some of that would want to be picked up by pact-js directly.

There is some really good discussion points on there. I like the notion of reading from package.json or a .pactrc file, as you say, it is much similar to other node projects.

I think we'd want to configure the jest plugin to clean out the pacts directory on start, then use update for the pact file write mode. This avoids keeping removed interactions around, while still allowing multiple test files to describe the same consumer/provider pairing.

Yeah we currently use update at work, along with using rimraf <rootDir>/pacts prior to running our tests, which works great. However there is additional considerations with watch mode.

If we have 4 pact tests for diff consumer/provider pairs, when we update a single pact test in watch mode, we would expect that only that file is re-executed by Jest. If we delete all the pacts, and only re-run one, we will be missing 3 of our consumer/provider pacts.

If we can find a way to only delete one pact file, that would work nicely, expect for the scenario where we have the same consumer/provider pair across multiple test files, as deleting the pact and running only one test, would again mean we are missing some information.

I think the best value is "what would the expected functionality be" - how can we help developers not have to think about the way pact is configured, and have it work for the majority of setups?

Agree fully on this point, and it is a tough nut to crack.

I think the main point, is not requiring developers to need to think about how pact needs to be configured before you can even start writing a test. It currently requires a reasonable amount of cognitive load, and it is easy to get the ordering wrong, and get into a muddle.

Making it work the majority of setups is really difficult as it is hard to assume what other peoples setups are, without seeing them / or getting feedback on the pain points in setting up pact, to see if we can absolve some of that. Wonder if it is worth asking for some input on Slack to get others thoughts.

With regards to the jsdom issue, I haven't come across that yet, but looking at the instructions, I assume that it is something we don't need to consider ourselves with, as the user can easily add the instruction to the top of their test file, or package.json

[YOU54F]

I've raised an proposal/featureRequest on the jest github repo, as looking at the docs, I am not sure if the watch plugin will cover our use cases. Thought it was worth asking the question :)

jestjs/jest#8522

[YOU54F]

Man I can't believe I wrote all that about 😂

Anyway it sounds like this might be doable now with a monkey patch see jestjs/jest#8714

[YOU54F]

Closing this down, as there was no real work here, just investigation. will add a note to the readme if anyone is interested to pick it up