diff --git a/docs/connector-development/README.md b/docs/connector-development/README.md index 5cef6c07115b..c54df6ca7fa8 100644 --- a/docs/connector-development/README.md +++ b/docs/connector-development/README.md @@ -6,16 +6,6 @@ Before building a new connector, review [Airbyte's data protocol specification]( If you need support along the way, visit the [Slack channel](https://airbytehq.slack.com/archives/C027KKE4BCZ) we have dedicated to helping users with connector development where you can search previous discussions or ask a question of your own. -### Connector vs. configured source vs. connection - -When building new connectors for Airbyte, it’s important to understand the difference between three related concepts: the connector, the configured source based on a connector, and the connection. - -**Connector**: A connector can be either a source or a destination. Usually, if you’re building a connection, you’re working with a source. The connector defines what’s required to access an API or a database such as protocol, URL paths to access, the way requests need to be structured, and how to extract records from responses. - -**Configured Source**: The configured source is what you set up when you provide the variables needed for the connector to access records. The exact fields of the configuration depend on the connector, but in most cases, it provides authentication information (username and password, API key) and information about which data to extract, for example, the start date to sync records from, a search query records have to match. - -**Connection**: A connection is an automated data pipeline that replicates data from a source to a destination. It links a configured source (based on a source connector) to a configured destination (based on a destination connector) to perform syncs. It defines things like the replication frequency (e.g. hourly, daily, manually) and which streams to replicate. - ### Process overview The first step in creating a new connector is to choose the tools you’ll use to build it. There are three basic approaches Airbyte provides to start developing a connector. To understand which approach you should take, review the [compatibility guide](./connector-builder-ui/connector-builder-compatibility.md). diff --git a/docs/connector-development/testing-connectors/README.md b/docs/connector-development/testing-connectors/README.md index 446905a1d6f4..aba8fed743d3 100644 --- a/docs/connector-development/testing-connectors/README.md +++ b/docs/connector-development/testing-connectors/README.md @@ -7,6 +7,20 @@ Multiple tests suites compose the Airbyte connector testing pyramid - [Connectors QA checks](https://docs.airbyte.com/contributing-to-airbyte/resources/qa-checks) - [Connector Acceptance tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference/) +## 🤖 CI + +If you want to run the global test suite, exactly like what is run in CI, you should install [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) and use the following command: + +```bash +airbyte-ci connectors --name= test +``` + +This will run all the tests for the connector, including the QA checks and the Connector Acceptance tests. +Connector Acceptance tests require connector configuration to be provided as a `config.json` file in a `.secrets` folder in the connector code directory. + +Our CI infrastructure runs the connector tests with [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md). Connectors tests are automatically and remotely triggered on your branch according to the changes made in your branch. +**Passing tests are required to merge a connector pull request.** + ## Connector specific tests ### 🐍 Python connectors @@ -20,6 +34,12 @@ poetry run pytest ### ☕ Java connectors +:::warning +Airbyte is undergoing a major revamp of the shared core Java destinations codebase, with plans to release a new CDK in 2024. +We are actively working on improving usability, speed (through asynchronous loading), and implementing [Typing and Deduplication](/using-airbyte/core-concepts/typing-deduping) (Destinations V2). +For this reason, Airbyte is not reviewing/accepting new Java connectors for now. +::: + We run Java connector tests with gradle. ```bash @@ -31,16 +51,3 @@ We run Java connector tests with gradle. Please note that according to the test implementation you might have to provide connector configurations as a `config.json` file in a `.secrets` folder in the connector code directory. -## 🤖 CI - -If you want to run the global test suite, exactly like what is run in CI, you should install [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) and use the following command: - -```bash -airbyte-ci connectors --name= test -``` - -This will run all the tests for the connector, including the QA checks and the Connector Acceptance tests. -Connector Acceptance tests require connector configuration to be provided as a `config.json` file in a `.secrets` folder in the connector code directory. - -Our CI infrastructure runs the connector tests with [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md). Connectors tests are automatically and remotely triggered on your branch according to the changes made in your branch. -**Passing tests are required to merge a connector pull request.** diff --git a/docs/connector-development/tutorials/building-a-java-destination.md b/docs/connector-development/tutorials/building-a-java-destination.md index d2cb3f1c0bcc..c5268647ede1 100644 --- a/docs/connector-development/tutorials/building-a-java-destination.md +++ b/docs/connector-development/tutorials/building-a-java-destination.md @@ -1,5 +1,13 @@ +--- +displayed_sidebar: docs +--- + # Building a Java Destination +:::warning +The template for building a Java Destination connector is currently unavailable. The Airbyte team is working on revamping the Java CDK. +::: + ## Summary This article provides a checklist for how to create a Java destination. Each step in the checklist diff --git a/docs/using-airbyte/core-concepts/readme.md b/docs/using-airbyte/core-concepts/readme.md index 6f7bb86ca0e5..e909d4723555 100644 --- a/docs/using-airbyte/core-concepts/readme.md +++ b/docs/using-airbyte/core-concepts/readme.md @@ -10,7 +10,7 @@ This page describes the concepts you need to know to use Airbyte. ## Source -A source is an API, file, database, or data warehouse that you want to ingest data from. +A source is an API, file, database, or data warehouse that you want to ingest data from. The configured source is what you set up when you provide the variables needed for the connector to access records. The exact fields of the configuration depend on the connector, but in most cases, it provides authentication information (username and password, API key) and information about which data to extract, for example, the start date to sync records from, a search query records have to match. ## Destination @@ -18,11 +18,11 @@ A destination is a data warehouse, data lake, database, or an analytics tool whe ## Connector -An Airbyte component which pulls data from a source or pushes data to a destination. +An Airbyte component which pulls data from a source or pushes data to a destination. A connector can be either a source or a destination. Usually, if you’re building a connection, you’re working with a source. The connector defines what’s required to access an API or a database such as protocol, URL paths to access, the way requests need to be structured, and how to extract records from responses. ## Connection -A connection is an automated data pipeline that replicates data from a source to a destination. Setting up a connection enables configuration of the following parameters: +A connection is an automated data pipeline that replicates data from a source to a destination. It links a configured source (based on a source connector) to a configured destination (based on a destination connector) to perform syncs. It defines things like the replication frequency (e.g. hourly, daily, manually) and which streams to replicate.Setting up a connection enables configuration of the following parameters: | Concept | Description | | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------ | diff --git a/docusaurus/sidebars.js b/docusaurus/sidebars.js index 51f1efa2d630..5bcc1e65987c 100644 --- a/docusaurus/sidebars.js +++ b/docusaurus/sidebars.js @@ -179,16 +179,15 @@ const sectionHeader = (title) => ({ const buildAConnector = { type: "category", - label: "Build a Connector", + label: "Building Connectors", + link: { + type: "doc", + id: "connector-development/README", + }, items: [ - { - type: "doc", - label: "Overview", - id: "connector-development/README", - }, { type: "category", - label: "Connector Builder", + label: "No-Code Connector Builder", items: [ "connector-development/connector-builder-ui/overview", "connector-development/connector-builder-ui/connector-builder-compatibility", @@ -209,10 +208,10 @@ const buildAConnector = { }, { type: "category", - label: "Low-code connector development", + label: "Low-Code CDK", items: [ { - label: "Low-code CDK Intro", + label: "Low-Code CDK Intro", type: "doc", id: "connector-development/config-based/low-code-cdk-overview", }, @@ -263,7 +262,7 @@ const buildAConnector = { { type: "category", - label: "Connector Development Kit", + label: "Python CDK", link: { type: "doc", id: "connector-development/cdk-python/README", @@ -276,6 +275,29 @@ const buildAConnector = { "connector-development/cdk-python/http-streams", "connector-development/cdk-python/python-concepts", "connector-development/cdk-python/stream-slices", + { + type: "category", + label: "Tutorials", + items: [ + "connector-development/tutorials/cdk-speedrun", + { + type: "category", + label: "Python CDK: Creating a Python Source", + items: [ + "connector-development/tutorials/custom-python-connector/getting-started", + "connector-development/tutorials/custom-python-connector/environment-setup", + "connector-development/tutorials/custom-python-connector/reading-a-page", + "connector-development/tutorials/custom-python-connector/reading-multiple-pages", + "connector-development/tutorials/custom-python-connector/check-and-error-handling", + "connector-development/tutorials/custom-python-connector/discover", + "connector-development/tutorials/custom-python-connector/incremental-reads", + "connector-development/tutorials/custom-python-connector/reading-from-a-subresource", + "connector-development/tutorials/custom-python-connector/concurrency", + ], + }, + + ], + }, ], }, { @@ -289,29 +311,6 @@ const buildAConnector = { "connector-development/testing-connectors/connector-acceptance-tests-reference", ], }, - { - type: "category", - label: "Tutorials", - items: [ - "connector-development/tutorials/cdk-speedrun", - { - type: "category", - label: "Python CDK: Creating a Python Source", - items: [ - "connector-development/tutorials/custom-python-connector/getting-started", - "connector-development/tutorials/custom-python-connector/environment-setup", - "connector-development/tutorials/custom-python-connector/reading-a-page", - "connector-development/tutorials/custom-python-connector/reading-multiple-pages", - "connector-development/tutorials/custom-python-connector/check-and-error-handling", - "connector-development/tutorials/custom-python-connector/discover", - "connector-development/tutorials/custom-python-connector/incremental-reads", - "connector-development/tutorials/custom-python-connector/reading-from-a-subresource", - "connector-development/tutorials/custom-python-connector/concurrency", - ], - }, - "connector-development/tutorials/building-a-java-destination", - ], - }, "connector-development/connector-specification-reference", "connector-development/schema-reference", "connector-development/connector-metadata-file", @@ -709,3 +708,10 @@ module.exports = { }, ], }; + +// Any temporarily archived content should be added here with a comment +// to indicate when it was archived and why. +// You can still view docs that are not linked to in the sidebar. + +// Java Destination template is not currently available for use +// "connector-development/tutorials/building-a-java-destination", \ No newline at end of file