airbyte-ci README cleanup part 2 (#36426)

## What Following up and adding more structure to airbyte-ci readme. No code changes. Following up #36239. ## Follow-up work - [ ] @alafanechere mentioned before — we should clean out `connector_ops` and move the pieces we need into `common_utils`, remove the package. - [ ] Rename `connectors_qa` into `connector_metadata_checks` - [ ] Clarify what exactly `qa-engine` is doing.
airbytehq · Mar 25, 2024 · 0474574 · 0474574
1 parent 70475d0
commit 0474574
Show file tree

Hide file tree

Showing 6 changed files with 57 additions and 52 deletions.
diff --git a/airbyte-ci/README.md b/airbyte-ci/README.md
@@ -7,8 +7,12 @@ The installation instructions for the `airbyte-ci` CLI tool cal be found here
 
 ## Tools
 
-| Directory                                     | Description                                                        |
-| --------------------------------------------- | ------------------------------------------------------------------ |
-| [`base_images`](connectors/base_images)       | A set of tools to build and publish Airbyte base connector images. |
-| [`ci_credentials`](connectors/ci_credentials) | A CLI tool to fetch connector secrets from GCP Secrets Manager.    |
-| [`connector_ops`](connectors/connector_ops)   | A python package with utils reused in internal packages.                         |
+| Directory                                          | Description                                                                                                                   |
+| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| [`base_images`](connectors/base_images)            | A set of tools to build and publish Airbyte base connector images.                                                            |
+| [`ci_credentials`](connectors/ci_credentials)      | A CLI tool to fetch connector secrets from GCP Secrets Manager.                                                               |
+| [`connector_ops`](connectors/connector_ops)        | A python package with utils reused in internal packages.                                                                      |
+| [`connectors_qa`](connectors/connectors_qa/)       | A tool to verify connectors have sounds assets and metadata.                                                                  |
+| [`metadata_service`](connectors/metadata_service/) | Tools to generate connector metadata and registry.                                                                            |
+| [`pipelines`](connectors/pipelines/)               | Airbyte CI pipelines, including formatting, linting, building, testing connectors, etc. Connector acceptance tests live here. |
+| [`qa_engine`](connectors/qa-engine/)               | Tools to fetch connector catalog metrics and parameters, such as adoption metrics.                                             |
diff --git a/airbyte-ci/connectors/README.md b/airbyte-ci/connectors/README.md
@@ -1,3 +1,6 @@
 # Airbyte Connectors CI
 
-This folder is a collection of systems, tools and scripts that are used to run CI/CD systems specific to our connectors.
+This folder is a collection of systems, tools and scripts that are used to run CI/CD systems
+specific to our connectors.
+
+For the list of tools and subfolders, please see [README in `airbyte-ci`](../README.md).
diff --git a/airbyte-ci/connectors/common_utils/README.md b/airbyte-ci/connectors/common_utils/README.md
@@ -0,0 +1,7 @@
+# Airbyte CI Common Utils
+
+`common_utils` is a Python package that provides common utilities that are used in other `airbyte-ci` tools, such as `ci_credentials` and `base_images`.
+
+Currently:
+- Logger
+- GCS API client
diff --git a/airbyte-ci/connectors/connectors_qa/README.md b/airbyte-ci/connectors/connectors_qa/README.md
@@ -1,20 +1,23 @@
 # Connectors QA
 
 This package has two main purposes:
-* Running QA checks on connectors.
-* Generating the QA checks documentation that are run on connectors.
+
+- Running assets and metadata verification checks on connectors.
+- Generating the QA checks documentation that are run on connectors.
 
 ## Usage
 
 ### Install
 
+Connectors QA is an internal Airbyte package that is not published to PyPI. To install it, run the
+following command from this directory:
+
 ```bash
 pipx install .
 ```
 
-This will make `connectors-qa` available in your `PATH`.
-
-Feel free to run `connectors-qa --help` to see the available commands and options.
+This will make `connectors-qa` available in your `PATH`. Run `connectors-qa --help` to see the
+available commands and options.
 
 ### Examples
 
@@ -24,6 +27,7 @@ Feel free to run `connectors-qa --help` to see the available commands and option
 # This command must run from the root of the Airbyte repo
 connectors-qa run --name=source-faker --name=source-google-sheets
 ```
+
 #### Running QA checks on all connectors:
 
 ```bash
@@ -63,15 +67,21 @@ poetry install
 ```
 
 ### Dependencies
+
 This package uses two local dependencies:
-* [`connector_ops`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/connector_ops): To interact with the `Connector` object.
-* [`metadata_service/lib`]((https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/metadata_service/lib)): To validate the metadata of the connectors.
+
+- [`connector_ops`](../connector_ops): To interact with the `Connector` object.
+- [`metadata_service/lib`](../metadata_service/lib): To validate the metadata of the connectors.
 
 ### Adding a new QA check
 
-To add a new QA check, you have to create add new class in one of the `checks` module. This class must inherit from `models.Check` and implement the `_run` method. Then, you need to add an instance of this class to the `ENABLED_CHECKS` list of the module.
+To add a new QA check, you have to create add new class in one of the `checks` module. This class
+must inherit from `models.Check` and implement the `_run` method. Then, you need to add an instance
+of this class to the `ENABLED_CHECKS` list of the module.
+
+**Please run the `generate-documentation` command to update the documentation with the new check and
+commit it in your PR.**:
 
-**Please run the `generate-documentation` command to update the documentation with the new check and commit it in your PR.**:
 ```bash
 # From airbyte repo root
 connectors-qa generate-documentation docs/contributing-to-airbyte/resources/qa-checks.md
@@ -98,14 +108,19 @@ poe lint
 ## Changelog
 
 ### 1.0.3
-Disable `CheckDocumentationStructure` for now. 
+
+Disable `CheckDocumentationStructure` for now.
 
 ### 1.0.2
+
 Fix access to connector types: it should be accessed from the `Connector.connector_type` attribute.
 
 ### 1.0.1
-* Add `applies_to_connector_types` attribute to `Check` class to specify the connector types that the check applies to.
-* Make `CheckPublishToPyPiIsEnabled` run on source connectors only.
+
+- Add `applies_to_connector_types` attribute to `Check` class to specify the connector types that
+  the check applies to.
+- Make `CheckPublishToPyPiIsEnabled` run on source connectors only.
 
 ### 1.0.0
+
 Initial release of `connectors-qa` package.
diff --git a/airbyte-ci/connectors/metadata_service/README.md b/airbyte-ci/connectors/metadata_service/README.md
@@ -1,12 +1,17 @@
 # Metadata Service
-This is the begining of metadata service for airbyte.
+
+Airbyte Metadata Service manages the Airbyte Connector Registry.
 
 This system is responsible for the following:
+
 - Validating Connector metadata
 - Storing Connector metadata in GCS
 - Serving Connector metadata to various consumers
 - Aggregating Connector metadata to provide a unified view of all connectors
 - Triggering actions based on changes to Connector metadata
 
 ## Subsystems
-- [Metadata Orchestrator](./orchestrator/README.md)
+
+- [Metadata Lib](./lib) responsible for preparing and validating connector metadata.
+- [Metadata Orchestrator](./orchestrator) responsible for gathering metadata into the registry,
+  using Dagster.
diff --git a/airbyte-ci/connectors/metadata_service/orchestrator/README.md b/airbyte-ci/connectors/metadata_service/orchestrator/README.md
@@ -4,40 +4,11 @@ This is the Orchestrator for Airbyte metadata built on Dagster.
 
 # Setup
 
-## Prerequisites
-
-#### Poetry
-
-Before you can start working on this project, you will need to have Poetry installed on your system.
-Please follow the instructions below to install Poetry:
-
-1. Open your terminal or command prompt.
-2. Install Poetry using the recommended installation method:
-
-```bash
-curl -sSL https://install.python-poetry.org | POETRY_VERSION=1.5.1 python3 -
-```
-
-Alternatively, you can use `pip` to install Poetry:
-
-```bash
-pip install --user poetry
-```
-
-3. After the installation is complete, close and reopen your terminal to ensure the newly installed
-   `poetry` command is available in your system's PATH.
-
-For more detailed instructions and alternative installation methods, please refer to the official
-Poetry documentation: https://python-poetry.org/docs/#installation
-
-### Using Poetry in the Project
-
-Once Poetry is installed, you can use it to manage the project's dependencies and virtual
-environment. To get started, navigate to the project's root directory in your terminal and follow
-these steps:
-
 ## Installation
 
+Metadata Orchestrator uses Poetry as dependency manager and build system. The guide below
+assumes you have Poetry installed.
+
 ```bash
 poetry install
 cp .env.template .env