Skip to content

Latest commit

 

History

History
 
 

building-new-connector

README.md

Developing Connectors

Airbyte supports two types of connectors: Sources and Destinations. A connector takes the form of a Docker image which follows the Airbyte specification.

To build a new connector in Java or Python, we provide templates so you don't need to start everything from scratch.

Note: you are not required to maintain the connectors you create. The goal is that the Airbyte core team and the community help maintain the connector.

The Airbyte specification

Before building a new connector, review Airbyte's data protocol specification.

Adding a new connector

Requirements

To add a new connector you need to:

  1. Implement & Package your connector in an Airbyte Protocol compliant Docker image
  2. Add integration tests for your connector. At a minimum, all connectors must pass Airbyte's standard test suite, but you can also add your own tests.
  3. Document how to build & test your connector

Each requirement has a subsection below.

1. Implement & package the connector

If you are building a connector in any of the following languages/frameworks, then you're in luck! We provide autogenerated templates to get you started quickly:

  • Python Source Connector
  • Singer-based Python Source Connector. Singer.io is an open source framework with a large community and many available connectors (known as taps & targets). To build an Airbyte connector from a Singer tap, wrap the tap in a thin Python package to make it Airbyte Protocol-compatible. See the Github Connector for an example of an Airbyte Connector implemented on top of a Singer tap.
  • Generic Connector: This template provides a basic starting point for any language.

Creating a connector from a template

Run the interactive generator:

cd airbyte-integrations/connector-templates/generator
npm install
npm run generate

and choose the relevant template. This will generate a new connector in the airbyte-integrations/connectors/<your-connector> directory.

If you are developing a Python/Singer connector, you may find the building a Python connector tutorial helpful.

2. Integration tests

At a minimum, your connector must implement the standard tests described in Testing Connectors

3. Document building & testing your connector

To merge your connector, Airbyte needs to know how to build & test it. If you're writing in Python or Java, skip this section -- it is provided automatically.

If you're writing in another language, please document the commands needed to:

  1. Build your connector docker image (usually this is just docker build . but let us know if there are necessary flags, gotchas, etc..)
  2. Run any unit or integration tests in a Docker image.

Your integration and unit tests must be runnable entirely within a Docker image. This is important to guarantee consistent build environments.

When you submit a PR to Airbyte with your connector, the reviewer will use the commands you provide to integrate your connector into Airbyte's build system as follows:

  1. :airbyte-integrations:connectors:source-<name>:build should run unit tests and build the integration's Docker image
  2. :airbyte-integrations:connectors:source-<name>:integrationTest should run integration tests including Airbyte's Standard test suite.

Best practices

Make sure to review the Best Practices for Connector Development guide. Following best practices is not a requirement for merging your contribution to Airbyte, but it certainly doesn't hurt ;)

Updating a connector

Once you've finished iterating on the changes to a connector as specified in its README.md, follow these instructions to tell Airbyte to use the latest version of your connector.

  1. Bump the version in the Dockerfile of the connector (LABEL io.airbyte.version=X.X.X).

  2. Update the connector version in:

    • STANDARD_SOURCE_DEFINITION if it is a source
    • STANDARD_DESTINATION_DEFINITION if it is a destination.

  3. Build the connector with the semantic version tag locally:

    ./tools/integrations/manage.sh build airbyte-integrations/connectors/<connector-name>

  4. Submit a PR containing the changes you made.

  5. One of Airbyte maintainers will review the change and publish the new version of the connector to Docker hub:

    ./tools/integrations/manage.sh publish airbyte-integrations/connectors/<connector-name>

  6. The new version of the connector is now available for everyone who uses it. Thank you!