From 0474574454a4a10f521a5b608ffe8b5dbd1c48b1 Mon Sep 17 00:00:00 2001 From: Natik Gadzhi Date: Mon, 25 Mar 2024 16:31:03 -0700 Subject: [PATCH] airbyte-ci README cleanup part 2 (#36426) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What Following up and adding more structure to airbyte-ci readme. No code changes. Following up #36239. ## Follow-up work - [ ] @alafanechere mentioned before — we should clean out `connector_ops` and move the pieces we need into `common_utils`, remove the package. - [ ] Rename `connectors_qa` into `connector_metadata_checks` - [ ] Clarify what exactly `qa-engine` is doing. --- airbyte-ci/README.md | 14 ++++--- airbyte-ci/connectors/README.md | 5 ++- airbyte-ci/connectors/common_utils/README.md | 7 ++++ airbyte-ci/connectors/connectors_qa/README.md | 39 +++++++++++++------ .../connectors/metadata_service/README.md | 9 ++++- .../metadata_service/orchestrator/README.md | 35 ++--------------- 6 files changed, 57 insertions(+), 52 deletions(-) create mode 100644 airbyte-ci/connectors/common_utils/README.md diff --git a/airbyte-ci/README.md b/airbyte-ci/README.md index f0ecc17a17a4..31ce08ac534c 100644 --- a/airbyte-ci/README.md +++ b/airbyte-ci/README.md @@ -7,8 +7,12 @@ The installation instructions for the `airbyte-ci` CLI tool cal be found here ## Tools -| Directory | Description | -| --------------------------------------------- | ------------------------------------------------------------------ | -| [`base_images`](connectors/base_images) | A set of tools to build and publish Airbyte base connector images. | -| [`ci_credentials`](connectors/ci_credentials) | A CLI tool to fetch connector secrets from GCP Secrets Manager. | -| [`connector_ops`](connectors/connector_ops) | A python package with utils reused in internal packages. | +| Directory | Description | +| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | +| [`base_images`](connectors/base_images) | A set of tools to build and publish Airbyte base connector images. | +| [`ci_credentials`](connectors/ci_credentials) | A CLI tool to fetch connector secrets from GCP Secrets Manager. | +| [`connector_ops`](connectors/connector_ops) | A python package with utils reused in internal packages. | +| [`connectors_qa`](connectors/connectors_qa/) | A tool to verify connectors have sounds assets and metadata. | +| [`metadata_service`](connectors/metadata_service/) | Tools to generate connector metadata and registry. | +| [`pipelines`](connectors/pipelines/) | Airbyte CI pipelines, including formatting, linting, building, testing connectors, etc. Connector acceptance tests live here. | +| [`qa_engine`](connectors/qa-engine/) | Tools to fetch connector catalog metrics and parameters, such as adoption metrics. | diff --git a/airbyte-ci/connectors/README.md b/airbyte-ci/connectors/README.md index 2b48853b0a44..0af1e828f1b8 100644 --- a/airbyte-ci/connectors/README.md +++ b/airbyte-ci/connectors/README.md @@ -1,3 +1,6 @@ # Airbyte Connectors CI -This folder is a collection of systems, tools and scripts that are used to run CI/CD systems specific to our connectors. \ No newline at end of file +This folder is a collection of systems, tools and scripts that are used to run CI/CD systems +specific to our connectors. + +For the list of tools and subfolders, please see [README in `airbyte-ci`](../README.md). diff --git a/airbyte-ci/connectors/common_utils/README.md b/airbyte-ci/connectors/common_utils/README.md new file mode 100644 index 000000000000..9565733d1069 --- /dev/null +++ b/airbyte-ci/connectors/common_utils/README.md @@ -0,0 +1,7 @@ +# Airbyte CI Common Utils + +`common_utils` is a Python package that provides common utilities that are used in other `airbyte-ci` tools, such as `ci_credentials` and `base_images`. + +Currently: +- Logger +- GCS API client diff --git a/airbyte-ci/connectors/connectors_qa/README.md b/airbyte-ci/connectors/connectors_qa/README.md index c441d6643ac5..b09b7781842d 100644 --- a/airbyte-ci/connectors/connectors_qa/README.md +++ b/airbyte-ci/connectors/connectors_qa/README.md @@ -1,20 +1,23 @@ # Connectors QA This package has two main purposes: -* Running QA checks on connectors. -* Generating the QA checks documentation that are run on connectors. + +- Running assets and metadata verification checks on connectors. +- Generating the QA checks documentation that are run on connectors. ## Usage ### Install +Connectors QA is an internal Airbyte package that is not published to PyPI. To install it, run the +following command from this directory: + ```bash pipx install . ``` -This will make `connectors-qa` available in your `PATH`. - -Feel free to run `connectors-qa --help` to see the available commands and options. +This will make `connectors-qa` available in your `PATH`. Run `connectors-qa --help` to see the +available commands and options. ### Examples @@ -24,6 +27,7 @@ Feel free to run `connectors-qa --help` to see the available commands and option # This command must run from the root of the Airbyte repo connectors-qa run --name=source-faker --name=source-google-sheets ``` + #### Running QA checks on all connectors: ```bash @@ -63,15 +67,21 @@ poetry install ``` ### Dependencies + This package uses two local dependencies: -* [`connector_ops`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/connector_ops): To interact with the `Connector` object. -* [`metadata_service/lib`]((https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/metadata_service/lib)): To validate the metadata of the connectors. + +- [`connector_ops`](../connector_ops): To interact with the `Connector` object. +- [`metadata_service/lib`](../metadata_service/lib): To validate the metadata of the connectors. ### Adding a new QA check -To add a new QA check, you have to create add new class in one of the `checks` module. This class must inherit from `models.Check` and implement the `_run` method. Then, you need to add an instance of this class to the `ENABLED_CHECKS` list of the module. +To add a new QA check, you have to create add new class in one of the `checks` module. This class +must inherit from `models.Check` and implement the `_run` method. Then, you need to add an instance +of this class to the `ENABLED_CHECKS` list of the module. + +**Please run the `generate-documentation` command to update the documentation with the new check and +commit it in your PR.**: -**Please run the `generate-documentation` command to update the documentation with the new check and commit it in your PR.**: ```bash # From airbyte repo root connectors-qa generate-documentation docs/contributing-to-airbyte/resources/qa-checks.md @@ -98,14 +108,19 @@ poe lint ## Changelog ### 1.0.3 -Disable `CheckDocumentationStructure` for now. + +Disable `CheckDocumentationStructure` for now. ### 1.0.2 + Fix access to connector types: it should be accessed from the `Connector.connector_type` attribute. ### 1.0.1 -* Add `applies_to_connector_types` attribute to `Check` class to specify the connector types that the check applies to. -* Make `CheckPublishToPyPiIsEnabled` run on source connectors only. + +- Add `applies_to_connector_types` attribute to `Check` class to specify the connector types that + the check applies to. +- Make `CheckPublishToPyPiIsEnabled` run on source connectors only. ### 1.0.0 + Initial release of `connectors-qa` package. diff --git a/airbyte-ci/connectors/metadata_service/README.md b/airbyte-ci/connectors/metadata_service/README.md index 35438f6dbf9c..1bcf9192a3f8 100644 --- a/airbyte-ci/connectors/metadata_service/README.md +++ b/airbyte-ci/connectors/metadata_service/README.md @@ -1,7 +1,9 @@ # Metadata Service -This is the begining of metadata service for airbyte. + +Airbyte Metadata Service manages the Airbyte Connector Registry. This system is responsible for the following: + - Validating Connector metadata - Storing Connector metadata in GCS - Serving Connector metadata to various consumers @@ -9,4 +11,7 @@ This system is responsible for the following: - Triggering actions based on changes to Connector metadata ## Subsystems -- [Metadata Orchestrator](./orchestrator/README.md) \ No newline at end of file + +- [Metadata Lib](./lib) responsible for preparing and validating connector metadata. +- [Metadata Orchestrator](./orchestrator) responsible for gathering metadata into the registry, + using Dagster. diff --git a/airbyte-ci/connectors/metadata_service/orchestrator/README.md b/airbyte-ci/connectors/metadata_service/orchestrator/README.md index ed1c3b44ade9..104264a4044a 100644 --- a/airbyte-ci/connectors/metadata_service/orchestrator/README.md +++ b/airbyte-ci/connectors/metadata_service/orchestrator/README.md @@ -4,40 +4,11 @@ This is the Orchestrator for Airbyte metadata built on Dagster. # Setup -## Prerequisites - -#### Poetry - -Before you can start working on this project, you will need to have Poetry installed on your system. -Please follow the instructions below to install Poetry: - -1. Open your terminal or command prompt. -2. Install Poetry using the recommended installation method: - -```bash -curl -sSL https://install.python-poetry.org | POETRY_VERSION=1.5.1 python3 - -``` - -Alternatively, you can use `pip` to install Poetry: - -```bash -pip install --user poetry -``` - -3. After the installation is complete, close and reopen your terminal to ensure the newly installed - `poetry` command is available in your system's PATH. - -For more detailed instructions and alternative installation methods, please refer to the official -Poetry documentation: https://python-poetry.org/docs/#installation - -### Using Poetry in the Project - -Once Poetry is installed, you can use it to manage the project's dependencies and virtual -environment. To get started, navigate to the project's root directory in your terminal and follow -these steps: - ## Installation +Metadata Orchestrator uses Poetry as dependency manager and build system. The guide below +assumes you have Poetry installed. + ```bash poetry install cp .env.template .env