Support external schemas such as OAS/Swagger, JSON Schema, GraphQL or .proto

[mefellows]

NOTE: this is a WIP request, but wanted to get my thoughts down and give the community an opportunity to view

"can I automatically generate tests from my Swagger?".

This question is so common, we usually retort by linking to https://docs.pact.io/faq/#can-i-generate-my-pact-file-from-something-like-swagger.

That article gets to the essence of the problem ("marking your own exam").

I've gone back and forth on this a bit, alternating between solution and problem too many times, so I'm going to first pose it here as a "how might we".

How might we make it easier to use alternative schemas in contract testing, whilst providing the same guarantees as Pact does today?

Current challenges
The main one (which may simply be a perceived one) is duplication in defining a "contract". For RESTful APIs, many prefer to define a contract with OAS/Swagger and I'm seeing JSON Schema trending (using tools like ajv to validate the requests match. Pact has a good way of serialising HTTP interactions, but is it desirable to follow a similar approach for non-HTTP systems?

Potential solutions (currently focussed on the OAS use case but also applies for the JSONSchema case, and I think also GraphQL)

Use Case 1: Simplifying authoring of consumer tests

1. Accept the schema in the pact test, and reduce the duplication in the Pact test authoring
2. Allow the broker to accept an "accompanying schema", or accept an annotated schema (e.g. adding Pact specific x-amples to OAS)

In JS this might look something like this. We can omit much of the additional detail in the request and response body/headers, assuming the request is easily disambiguated.

const provider = new PactV3({
  consumer: "Matching Service V3",
  provider: "Animal Profile Service V3",
  schema: "/path/to/oas.yml",
})

provider
  .given("is not authenticated")
  .get("/animals/available")
  .responds(200)

    /// ... execute the test as usual

The mock server would be responsible for serving up an API that responds as defined in the OAS definition, and merge any Pact specific properties as discussed aboe.

For a provider test, it will look much the same as it does today, except that the matching rules will use the swagger definition.

Use Case 2: Provider driven contracts

1. Provider: OAS is authored with annotations for the example specifications (e.g. via x-examples or whatever the new extensions format is)

2. Provider: Pact reads the annotated swagger file and performs the verification much as it would if it instead had a Pact file

3. Provider: Publishes to the broker

4. Consumer: Writes a test that imports this OAS document

   const provider = new PactV3({
     consumer: "Matching Service V3",
     provider: "Animal Profile Service V3",
     schema: "/path/to/oas.yml", // <- more likely from the broker
   })
   
   // this setup just needs enough to discriminate the path in the OAS to extract the sample
   provider
     .given("is not authenticated")
     .get("/animals/available")
     .responds(200)
   
   // ... make the API call

5. Test only passes if all of the requests the consumer makes are compatible with the OAS document

6. Publish results to broker

Additional benefits:

• Conceptually, there are just two differences: it's just an "interface" over a Pact file, and it's provider initiated
• Reduces duplication when the two tools are used
• Provider never gets surprises about new provider states - because they define them in advance anyway (basically, the provider has full control over the data it cares about. We may also want to allow the consumer to override the specific examples - so long as they are compliant with the matching rules - to ensure that data changes on the provider don't impact tests. This would potentially help with flakiness)
• Retrofit use cases: it would open up the possibility to generate an OAS (or another schema) based on an existing functional test suite. For example, via a recording proxy or conversion from another format (e.g. Postman)
• We'd still know which bits of the OAS was actually being used by consumers
• You could do static analysis on the OAS vs Examples to identify functional / API path test gaps

Further reading:

GraphQL Metadata/Decorators

• https://www.apollographql.com/blog/graphql-schema-decorators-3c01a445b628/
• Expose user-defined meta-information via introspection API in form of directives graphql/graphql-spec#300
• AsyncAPI

JSON Schema

• https://json-schema.org/ (you don't need to do anything special to define metadata, so it could easily be either embedded into a Pact document or vice versa!)

[Lakitna]

I would like to raise the concept of signing contracts as use case 3. I'm planning to do a full writeup in the future as part of my article series on contract-based testing. I'll quickly write up the main concept for now.

For contract signing, we need both consumer-driven contracts (CDC) and a provider-driven contract (PDC). We can validate one contract with the other.

For example, the provider uploads a new PDC to the contract signer. The signer grabs all accompanying CDCs. It grabs all interactions from the CDCs and validates them with the PDC. The PDC is rejected if during this validation any error is thrown. If no errors are thrown the PDC is signed and stored in the central contract repository.

This also works when the consumer uploads a new contract: The consumer uploads a new CDC to the contract signer. The signer grabs the accompanying PCD and all interactions from the CDC. It validated the consumer's interactions against the PDC. The CDC is rejected if during this validation any error is thrown. If no errors are thrown the CDC is signed and stored in the central contract repository.

Main advantages:

• The provider will encounter a lot fewer issues with bad contracts. They are rejected before the provider even sees them.
• Faster feedback for both provider and consumer. Especially for common mistakes.
• Easier to understand conceptually: Everyone writes a contract.
• Opens up possibilities for more advanced reporting. For example, the provider could get automated reports telling them which parts of their interface are used most often.

Main disadvantages:

• The contract signer is central infrastructure -> is a dependency and a single point of failure.
• Easy to skip the CDC testing on the provider-side (which can also be an advantage in specific situation)

[bethesque]

I think this is a great idea, however, I think it needs a workflow to allow consumers to request functionality that doesn't yet exist. That's part of the "consumer driven" nature of "consumer driven contracts".

[Lakitna]

I agree.

There also has to be a way for the provider to mark things deprecated. This is already possible in OpenAPI Schema. Such a deprecation marker should raise a warning on the consumer-side.

In any case, the pattern as described above is pretty bare bones. But I think the concept is sound. I'm sure we can find solutions to the things I haven't thought about. We probably can reuse concepts that already exist in Pact. For requesting functionality, I'm specifically thinking about tagged and/or pending Pacts.

I already offered to @mefellows to talk about this with you folks. You're at +8 hours from me, but I'm sure we can find a moment to get into the nitty-gritty details :)

[alastairs]

Something else to throw into the mix, with implications for MessagePact, is AsyncAPI and any associated alternatives. It fulfils the same role as Open API Specifications/Swagger for events in a distributed system, and so has the same "making your own homework" caveats, but also probably many of the same benefits too.

Update 14 September 2020

I've also come across the Cloud Events specification which supports HTTP transports as well as AMQP and MQTT. @mefellows how would an HTTP-based message-passing spec work with Pact? E.g. Azure Event Grid.

[mefellows]

Good call out @alastairs. I'll update the comment above, because it's been on my (a) list for ages.

[Lakitna]

I'm just wondering. What are the next steps for this? Should we start expanding concepts? Which approach should we focus on? Do we need to implement a proof of concept?

Or am I trying to go too fast? 😄

[bethesque]

We should have something for you to try out in Pactflow in a few weeks.

[mefellows]

@alastairs sorry I missed your update, it doesn't send notifications for them.

As for cloud events, thanks for the share - I wasn't actually aware of this.

In my mind, this is almost a payload specific thing. The cloud events spec is fairly broad, so trying to do too much cloudevent specific stuff in Pact is probably a bad idea (there will be better tools for testing cloudevent-cy things over time). Where we do care about stuff is in the payload itself - Pact takes a fairly similar stance with respect to being protocol agnostic (although we don't currently have the concept of an "extension" - more on this in just a moment).

What's missing is the ability to differentiate between request/response, request only, and streaming variations of our (see my proposal in #71). Being able to do this, across different protocols (HTTP, message and whatever else we come up with) I think will do enough to cover this case.

More broadly, our issue is that we don't have a good mechanism to mix and match a protocol, transport and interaction style. e.g. we don't current have a way to to support a message based, AVRO formatted, request/response interaction.

Getting there is going to be a challenge, for various reasons, but i believe it's possible.

I'm going to create a separate issue that tracks this, because whilst it overlaps with this, I believe the intent is different.

[mefellows]

FYI pactflow/roadmap#4 is ready for developer preview. You can find a link on https://github.com/pactflow/roadmap/ to join our developer preview program to get a briefing on an early version of this feature.

[basickarl]

Hello! I'll just chip in a use-case that I would like regarding provider driven contracts. In my ideal world, I would like a 3rd party service API to provide a contract (json file hosted somewhere) that they have verified against their API. I would like to be able to download this contract and then verify my application against it. Pretty much the consumer-driven approach but the other way around.

We are trying to verify that our applications will work correctly with 3rd party services.

Creating that contract file from a swagger/openai is also a step in the right dorection!

[mefellows]

Whilst not a Pact specific feature, we do have a workflow similar to described in Pactflow: https://docs.pactflow.io/docs/bi-directional-contract-testing.

Upload OAS as a provider contract (ideally verified to be valid, but in the case of a 3rd party you'll just have to trust they have tested their API), generate a consumer contract and Pactflow will ensure they stay compatible.