Skip to content

Commit

Permalink
Add versioning documentation (#19738)
Browse files Browse the repository at this point in the history
* On Deploying versioning error from the bootloader

* Add ProtocolVersion description to airbyte protocol

* Add general protocol versioning doc
  • Loading branch information
gosusnp authored Nov 28, 2022
1 parent 63ae0ea commit 43e6a52
Show file tree
Hide file tree
Showing 3 changed files with 62 additions and 0 deletions.
23 changes: 23 additions & 0 deletions docs/troubleshooting/on-deploying.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,3 +95,26 @@ ERROR: Head "https://registry-1.docker.io/v2/airbyte/init/manifests/{XXX}": unau

You are most likely logged into Docker with your email address instead of your Docker ID.
Log out of Docker by running `docker logout` and try running `docker-compose up` again.

## Protocol Version errors from the bootloader when trying to upgrade

When starting up Airbyte, the bootloader may fail with the following error:
```
Aborting bootloader to avoid breaking existing connection after an upgrade. Please address airbyte protocol version support issues in the connectors before retrying.
```

We aborted the upgrade to avoid breaking existing connections due to a deprecation of protocol version.

Looking at the `airbyte-bootloader` logs, there should be a few messages describing the change of support range of the Airbyte Protocol:

```
2022-11-21 22:07:20 INFO i.a.b.ProtocolVersionChecker(validate):81 - Detected an AirbyteProtocolVersion range change from [0.0.0:2.0.0] to [1.0.0:2.0.0]
2022-11-21 22:07:20 WARN i.a.b.ProtocolVersionChecker(validate):98 - The following connectors need to be upgraded before being able to upgrade the platform
2022-11-21 22:07:20 WARN j.u.s.ReferencePipeline$3$1(accept):197 - Source: d53f9084-fa6b-4a5a-976c-5b8392f4ad8a: E2E Testing: protocol version: 0.2.1
```

From this example, this upgrade will drop the support for the major version 0 of the Airbyte Protocol. One connector here is problematic, we have the `E2E Testing` source connector that is blocking the upgrade because it is still using protocol version 0.2.1.

In order to resolve this situation, all the problematic connectors must be upgraded to a version that is using a newer version of the Airbyte Protocol. In this specific example, we should target version 1 or 2, our recommendation is to always target the most recent version.

For more details on how to upgrade a custom connector, you may want to refer to the [custom connector](../operator-guides/using-custom-connectors.md) documentation.
38 changes: 38 additions & 0 deletions docs/understanding-airbyte/airbyte-protocol-versioning.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Airbyte Protocol Versioning

## Goal

The goal of this document is to define our approach to protocol changes.

We need a compromise between frequent breaking changes that are heavy on operations and infinite backward compatibility which is a burden from a software engineering point of view.

## Versioning Scheme

We are using a `<MAJOR>.<MINOR>.<PATCH>` scheme for the Protocol Versioning. (see [SemVer](https://semver.org/)).

We increment the
* MAJOR version when you make incompatible protocol changes
* MINOR version when you add functionality in a backwards compatible manner
* PATCH version when you make backwards compatible bug fixes

## Development Guidelines

1. We will continue to do our best effort to avoid introducing breaking changes to the Airbyte Protocol.
2. When introducing a new minor version of the Airbyte Protocol, new fields must come with sensible defaults for backward compatibility within the same major version, or be entirely optional.
3. When introducing a new major version of the Airbyte Protocol, all connectors from the previous major version will continue to work. This requires the ability to “translate” messages between 1 major version of the Airbyte Protocol.

## Safeguards

To ensure continuous operation, we have a few safeguards to prevent breaking existing configuration through protocol version incompatibilities.

### When upgrading Airbyte

When removing support for older versions of the Protocol, there is a risk removing the support for a version that is currently used.

To mitigate this, as part of the pre-upgrade checks that happens in the `airbyte-bootloader`, we verify that any connector currently part of an active connection will still be supported after the upgrade.

If any connector fails this check, we abort the upgrade and the `airbyte-bootloader` logs contains a list of connectors to upgrade. Those connectors will need to be upgraded from the UI before the platform itself can be upgraded.

### When upgrading a Connector

When upgrading a Connector from the UI, we will verify that the Protocol Version is supported before finalizing the Connector upgrade.
1 change: 1 addition & 0 deletions docs/understanding-airbyte/airbyte-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ The `connectionSpecification` is [JSONSchema](https://json-schema.org) that desc

The specification also contains information about what features the Actor supports.

- `protocol_version` describes which version of the protocol the Connector supports. The default value is "0.2.0".
- `supported_destination_sync_modes` - describes which sync modes a destination is able to support. See [Sync Modes](#source-sync-modes).

`documentationUrl` and `changelogUrl` are optional fields that link to additional information about the connector.
Expand Down

0 comments on commit 43e6a52

Please sign in to comment.