Vervet is an HTTP API version lifecycle management tool, allowing APIs to be designed, developed, versioned and released from resources independently and concurrently.
In a large organization, there might be many teams involved in delivering a large API -- such as at Snyk where Vervet was developed.
Within a single small team, there is still often a need to simultaneously try new things in parts of an API while maintaining stability.
While Vervet was developed in the context of a RESTful API, Vervet can be used with any HTTP API expressed in OpenAPI 3 -- even if it does not adhere to strict REST principles.
To summarize the API versioning supported by Vervet:
Resource versions are defined in OpenAPI 3, as if each resource were a standalone service.
Resources are organized in a standard directory structure by release date, using OpenAPI extensions to define lifecycle concepts like stability.
- Resources are versioned independently by date and stability, with a well-defined deprecation and sunsetting policy.
- Additive, non-breaking changes can be made to released versions. Breaking changes trigger a new version.
- New versions deprecate and sunset prior versions, on a timeline determined by the stability level.
Read more about API versioning.
A brief tour of Vervet's features.
Vervet collects the OpenAPI specification of each resource version and merges them into a series of OpenAPI specifications that describe the entire application, at each distinct release version in its underlying parts.
Given a directory structure of resource versions, each defined by an OpenAPI specification as if it were an independent service:
tree resources
resources
├── _examples
│ └── hello-world
│ ├── 2021-06-01
│ │ └── spec.yaml
│ ├── 2021-06-07
│ │ └── spec.yaml
│ └── 2021-06-13
│ └── spec.yaml
└── projects
└── 2021-06-04
└── spec.yaml
and a Vervet project configuration that instructs how to put them together:
cat .vervet.yaml
apis:
my-api:
resources:
- path: "resources"
output:
path: "versions"vervet build aggregates these resources' individual OpenAPI specifications to describe the entire service API at each distinct version date and stability level from its component parts.
tree versions
versions/
├── 2021-06-01
│ ├── spec.json
│ └── spec.yaml
├── 2021-06-01~beta
│ ├── spec.json
│ └── spec.yaml
├── 2021-06-01~experimental
│ ├── spec.json
│ └── spec.yaml
├── 2021-06-04
│ ├── spec.json
│ └── spec.yaml
├── 2021-06-04~beta
│ ├── spec.json
│ └── spec.yaml
├── 2021-06-04~experimental
│ ├── spec.json
│ └── spec.yaml
├── 2021-06-07
│ ├── spec.json
│ └── spec.yaml
├── 2021-06-07~beta
│ ├── spec.json
│ └── spec.yaml
├── 2021-06-07~experimental
│ ├── spec.json
│ └── spec.yaml
├── 2021-06-13
│ ├── spec.json
│ └── spec.yaml
├── 2021-06-13~beta
│ ├── spec.json
│ └── spec.yaml
└── 2021-06-13~experimental
├── spec.json
└── spec.yaml
From 2024-10-15, Vervet introduced a new "simplified versioning" scheme.
The main differences introduced by simplified versioning are:
-
Stability dropped from the spec level to the individual path level: Each API path can now be labeled as either
betaorGA(general availability). Theexperimentalstability level has been removed. This means that paths within the same version of the spec can have different stability statuses, allowing for greater flexibility and precision. -
No changes required from developers defining APIs: Developers still define their APIs in the same way as before, and Vervet handles the collation and versioning of specs in the new format automatically. The way specs are defined, updated, and maintained remains familiar, ensuring a smooth transition.
-
End-user API requests are simplified: API consumers no longer need to specify the stability when calling a particular version. They can simply provide the date of the version they want, and Vervet will resolve whether each path is
GAorbeta. If a path has been promoted toGA, subsequent versions cannot publish it asbetaagain.
This change is aimed at simplifying how users interact with versioned APIs while maintaining clear definitions and stabilities at the path level.
To illustrate how simplified versioning works in practice, let’s consider some examples:
Consider an API with paths /pets and /owners. Suppose the /pets path is GA and the /owners path is beta as of version date 2024-10-20.
-
Request:
GET /api/2024-10-20/pets- Response: This request will be handled by the GA version of
/pets.
- Response: This request will be handled by the GA version of
-
Request:
GET /api/2024-10-20/owners- Response: This request will be handled by the beta version of
/owners.
- Response: This request will be handled by the beta version of
The user does not need to specify the stability explicitly. Vervet determines the appropriate path stability (GA or beta) automatically.
Let’s say that as of 2024-11-10, the /owners path is promoted to GA.
- Request:
GET /api/2024-11-15/owners- Response: The request will be handled by the GA version of
/owners.
- Response: The request will be handled by the GA version of
Once a path is promoted to GA, any subsequent version cannot publish it as beta again. Therefore, users can be confident that they are always accessing a stable version of a path if it is marked as GA.
Consider an API version dated 2024-12-01 with the following paths:
-
/pets: GA -
/owners: GA -
/appointments: beta -
Request:
GET /api/2024-12-01/appointments- Response: The request will be handled by the beta version of
/appointments.
- Response: The request will be handled by the beta version of
-
Request:
GET /api/2024-12-01/pets- Response: The request will be handled by the GA version of
/pets.
- Response: The request will be handled by the GA version of
This approach allows different parts of an API to evolve at different paces, providing flexibility for developers and a clearer experience for end-users.
Since Vervet models the composition, construction and versioning of an API, it is well positioned to coordinate code and artifact generation through the use of templates.
Generators may be defined in a YAML file, such as generators.yaml:
generators:
version-readme:
scope: version
filename: "{{ .Path }}/README"
template: "{{ .Here }}/templates/README.tmpl" # Located relative to the location of generators.yamlThe context of README.tmpl has full access to the resource version metadata and OpenAPI document object model.
Generated by vervet. DO NOT EDIT!
# My API
Files in this directory were generated by `@snyk/vervet`
for resource `{{ .ResourceVersion.Name }}` at version `{{ .ResourceVersion.Version.String }}`.In a project with a .vervet.yaml configuration, execute the generators with
vervet generate -g generators.yaml
The simple generator above produces a README in each resource version directory.
tree resources
resources
└── thing
└── 2021-10-21
├── README
└── spec.yaml
Generators are defined using Go templates.
Template syntax may also be used to express a directory structure of many files. A more advanced example, an Express controller generated from each operation in a resource version OpenAPI spec:
generators:
version-controller:
scope: version
# `files:` generates a collection of files -- which itself is expressed as a
# YAML template. Keys in this YAML are the paths of the files to generate,
# whose values are the file contents.
files: |-
{{- $path := .Path -}}
{{- $resource := .ResourceVersion -}}
{{- $version := .ResourceVersion.Version -}}
{{- range $path, $pathItem := .ResourceVersion.Document.Paths -}}
{{- range $method, $operation := $pathItem -}}
{{- $operationId := $operation.operationId -}}
{{/* Construct a context object using the 'map' function */}}
{{- $ctx := map "Context" . "OperationId" $operationId }}
{{ $path }}/{{ $operationId }}.ts: |-
{{/*
Evaluate the template by including it with the necessary context.
The generator's template (controller.ts.tmpl) is included as
"contents" from within the `files:` template.
*/}}
{{ include "contents" $ctx | indent 2 }}
{{ end }}
{{- end -}}
template: "{{ .Here }}/templates/controller.ts.tmpl"In this case, a template is being applied per operationId in the spec.yaml generated in the prior step. version-controller produces a collection of files, a controller module per resource, per version, per operation.
Finally, a note on scoping. Generators can be scoped to either a version or a resource.
scope: version generator templates execute with VersionScope. This maps 1:1 with a single resource version OpenAPI specification.
scope: resource generator templates execute with ResourceScope. This is a collection of resource versions, useful for building resource routers.
Within a project:
npm install @snyk/vervet
Or installed globally:
npm install -g @snyk/vervet
NPM packaging adapted from https://github.com/manifoldco/torus-cli.
Go >= 1.16 required.
go install github.com/snyk/vervet/v6/cmd/vervet@latest
Building from source locally:
go build ./cmd/vervet
or
make build
Vervet uses a reference set of OpenAPI documents in testdata/resources in
tests. CLI tests compare runtime compiled output with pre-compiled, expected
output in testdata/output to detect regressions.
When introducing changes that intentionally change the content of compiled output:
- Run
go generate ./testdatato update the contents oftestdata/output - Verify that the compiled output is correct
- Commit the changes to
testdata/outputin your proposed branch
A new version of vervet will automatically be generated for Github and npm when new features
are introduced, i.e. when commits are merged that are marked with feat:.
After removing the endpoint version code and specs, you may see this issue:
ENOENT: no such file or directory, open '.../spec.yaml'
To solve this:
- Temporarily ignore the endpoint version code in
.vervet.yaml - Remove the endpoint versions from
catalog-info.yaml - Remove the old OpenAPI specs.
In order to understand why Vervet Underground exists and the problem it solves, you should first become familiar with the API versioning scheme that Vervet supports. The main idea is, an API may be authored in parts, each of those parts may be versioned, and all the distinct versions are assembled to produce a cohesive timeline of versions for the entire service.
Just as Vervet compiles a timeline of OpenAPI versions for a single service from independently versioned parts, Vervet Underground (VU) compiles a timeline of OpenAPI spec versions for a SaaS from independently versioned microservices, each of which contributes parts of the SaaS API.
To illustrate how this works in practice, let's deconstruct a pet store into two services:
petfood.default.svc.cluster.local, which knows about pet food.animals.default.svc.cluster.local, which knows about animals.
For sake of example, let's assume the following versions are published by each service:
petfood has:
2021-07-04~experimental2021-08-09~beta2021-08-09~experimental(beta released, with some parts still experimental, so both are published)2021-09-14. (first GA)2021-09-14~beta2021-09-14~experimental
animals has:
2021-09-10~experimental2021-10-04~experimental2021-10-12~beta2021-10-12~experimental2021-11-052021-11-05~beta2021-11-05~experimental
And, the OpenAPI spec for each version is available at /openapi. /openapi provides a JSON array of OpenAPI versions supported by the service, and /openapi/{version} fetches the OpenAPI spec for that particular {version}. For example, GET http://petfood.default.svc.cluster.local/openapi/2021-09-14~experimental.
There is some nuance to this to be aware of. You'll notice that some dates have multiple versions with different stabilities. This can happen because on that date, there is more than one API version available at different stability levels.
There are also some assumptions. These services are cooperatively contributing to the public pet store SaaS API. They cannot conflict with each other -- no overwriting each other's OpenAPI components, or publishing conflicting paths.
From these service versions, what versions of the pet store API are published to the public SaaS consumer? Well, the union of all of them! So we should be able to enumerate these versions from the public SaaS API:
GET https://api.petstore.example.com/openapi
200 OK
Content-Type: application/json
[
`2021-07-04~experimental`, // Contains only petfood so far...
`2021-08-09~experimental`,
`2021-08-09~beta`,
`2021-09-10~beta` // Petfood only (no beta version of animals yet...)
`2021-09-10~experimental`, // First animals (experimental version) + petfood
`2021-09-14`, // Petfood GA only, animals isn't GA yet
`2021-09-14~beta`,
`2021-09-14~experimental`,
`2021-10-04`,
`2021-10-04~beta`,
`2021-10-04~experimental`,
`2021-10-12`,
`2021-10-12~beta`,
`2021-10-12~experimental`,
`2021-11-05`, // First GA release of animals, also has petfood GA (from most recent 2021-09-14)
`2021-11-05~beta`,
`2021-11-05~experimental`,
]
The examples so far have been kept simple by assuming released API versions do not change. In practice, non-breaking changes are allowed to be made to existing versions of the API at any time. Non-breaking changes must be additive and optional. It is fine to add new HTTP methods, endpoints, request parameters or response fields, so long that the added parameters or fields are not required -- which existing generated client code would have no way of knowing about.
With VU, we should be able to request a version of the public SaaS API as it was on a given date, regardless of the version release date.
Let's assume that on 2021-11-08, the petfood service team adds a PATCH method to all their resources, to allow existing orders to be modified before they ship. It's a reasonable thing to do -- a new HTTP method doesn't break existing behavior! The team adds this method to every active (non-sunset) version retroactively -- why not? it was essentially the same backend code to implement it!
Vervet Underground not only compiles the initially published APIs from component services, it tracks changes in these APIs and updates its SaaS-level view of the API accordingly. So, VU scrapes the /openapi endpoints of its services periodically and detects the API changes on 2021-11-08, even though no new API was explicitly released that day, and now represents it as a new "discovered" version:
GET https://api.petstore.example.com/openapi
200 OK
Content-Type: application/json
[
`2021-07-04~experimental`,
...
`2021-11-05`,
`2021-11-05~beta`,
`2021-11-05~experimental`,
`2021-11-08`, // A wild new version appears!
`2021-11-08~beta`,
`2021-11-08~experimental`,
]
VU scrapes the /openapi endpoints of each service and tracks the changes in each version found. This may be stored in a directory structure, such as:
services/petfood
├── 2021-09-14~experimental
│ ├── 2021-09-14_11_23_24.spec.yaml
│ └── 2021-11-08_13_14_15.spec.yaml
...
where the scraped OpenAPI is compared against the most recent last snapshot: if they differ, a new snapshot is taken.
When the new 2021-11-08 snapshot is detected, this triggers a rebuild of the top-level SaaS OpenAPI specs with that new version added. The arrow of time eventually flows only one way and storage is cheap, so it's assumed that the compiled OpenAPI specs will be statically compiled up-front as service API changes are detected.
This snapshot version should not be taken to represent a breaking-change release, which has different deprecation and sunsetting implications. It is only used to represent what the API looked like at a given point in time.
If a version date prior to 2021-11-08 resolves to 2021-09-14~experimental, you should see the 2021-09-14~experimental contributions to the API as it would have appeared at that time, in other words, you should see a view based on the 2021-07-04_11_23_24.spec.yaml snapshot of 2021-09-14~experimental.
If a version date after 2021-11-08 matches 2021-09-14~experimental, let's say a request for 2021-12-10~experimental, then you should see it as it would appear after the non-breaking change, 2021-11-08_13_14_15.spec.yaml.
VU aggregates OpenAPI specs from multiple services and serves them up in a single place for:
- Docs
- Docs will likely either render public
/openapidirectly client-side or periodically scrape & update
- Docs will likely either render public
- Routing public v3
/openapito VU's aggregated OpenAPI versions - Add Akamai configuration to serve the blockstored specs initially for
/openapi- Formal frontend presentation will be later
This unblocks docs for multi-service decomp.
- Could use block storage with a history of changes per service per version
- Static config that tells VU where to scrape upstream OpenAPI from services (registry and friends)
- Cron job to periodically scrape and update
- Or we can try to make this push and set up a webhook...
From 2024-10-15, Vervet Underground also integrates the simplified versioning model. With simplified versioning:
- API versions compiled by VU no longer require consumers to specify the stability level (
beta,GA) in the request. Instead, consumers simply request a version by date, and the paths within that version automatically resolve to their correct stability levels (betaorGA). - Experimental stabilities are no longer supported; all paths are now either
betaorGA. - Changes to a path's stability are reflected by date, ensuring consistent access to
GAorbetaversions without ambiguity.
This results in a more straightforward API usage for clients, as they no longer need to be concerned with explicitly requesting stabilities, which reduces friction and simplifies interaction with the aggregated service APIs.
To better understand how simplified versioning works in comparison to the previous versioning model, let’s consider an example both before and after the pivot date of 2024-10-15.
Consider a pet store API with services for animals and petfood. Assume the following versions:
-
animalshas:2024-10-01~beta2024-10-01~experimental
-
petfoodhas:2024-09-20~GA2024-10-01~beta
If a client requested:
-
Request:
GET /api/2024-10-01~beta/animals- Response: This request would be served by the beta version of
/animals.
- Response: This request would be served by the beta version of
-
Request:
GET /api/2024-10-01~experimental/animals- Response: This request would be served by the experimental version of
/animals.
- Response: This request would be served by the experimental version of
-
Request:
GET /api/2024-09-20~GA/petfood- Response: This request would be served by the GA version of
/petfood.
- Response: This request would be served by the GA version of
Here, users must explicitly specify the stability (~beta, ~experimental, or ~GA), which adds complexity to the request.
After the pivot date of 2024-10-15, the simplified versioning model takes effect, where all paths are collated into a single spec with individual stabilities (beta or GA). Consider the following scenario:
-
animalshas:- Path
/animalsmarked asbetaon2024-10-20.
- Path
-
petfoodhas:- Path
/petfoodmarked asGAon2024-10-20.
- Path
If a client requested:
-
Request:
GET /api/2024-10-20/animals- Response: This request will be handled by the beta version of
/animalswithout specifying the stability explicitly.
- Response: This request will be handled by the beta version of
-
Request:
GET /api/2024-10-20/petfood- Response: This request will be handled by the GA version of
/petfood.
- Response: This request will be handled by the GA version of
With the simplified model, clients simply provide the date, and VU handles the rest, resolving the correct stability for each path automatically.
From 2024-10-15, Vervet Underground also integrates the simplified versioning model. With simplified versioning:
- API versions compiled by VU no longer require consumers to specify the stability level (
beta,GA) in the request. Instead, consumers simply request a version by date, and the paths within that version automatically resolve to their correct stability levels (betaorGA). - Experimental stabilities are no longer supported; all paths are now either
betaorGA. - Changes to a path's stability are reflected by date, ensuring consistent access to
GAorbetaversions without ambiguity.
This results in a more straightforward API usage for clients, as they no longer need to be concerned with explicitly requesting stabilities, which reduces friction and simplifies interaction with the aggregated service APIs.