Connector builder: Restructure documentation (#25353)

* authentication documentation

* some fixes

* fix broken link

* remove session token auth

* review comments

* first version of tutorial

* Revert "first version of tutorial"

This reverts commit bbdeb99.

* first version of tutorial

* review comments

* improvements

* partitioning documentation

* incremental sync

* review comments

* improvements

* record processing doc

* improvements

* error handler connector builder ui documentation

* add missing link

* overview page

* mention current limitation

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

airbyte-cdk/python/airbyte_cdk/connector_builder/README.md

@@ -1,6 +1,6 @@
 # Connector Builder Backend
 
-This is the backend for requests from the [Connector Builder](https://docs.airbyte.com/connector-development/config-based/connector-builder-ui/).
+This is the backend for requests from the [Connector Builder](https://docs.airbyte.com/connector-development/connector-builder-ui/overview/).
 
 ## Local development
 

docs/connector-development/README.md

@@ -8,9 +8,13 @@ To build a new connector in Java or Python, we provide templates so you don't ne
 
 Airbyte provides some Connector Development Kits (CDKs) to help you build connectors.
 
+### Connector builder UI
+
+The [connector builder UI](connector-builder-ui/overview.md) is based on the low-code development framework below and allows to develop and use connectors without leaving the Airbyte UI (no local development environment required).
+
 ### Low-code Connector-Development Framework
 
-You can use the [low-code framework](config-based/low-code-cdk-overview.md) to build source connectors for REST APIs via a [connector builder UI](config-based/connector-builder-ui.md) or by modifying boilerplate YAML files.
+You can use the [low-code framework](config-based/low-code-cdk-overview.md) to build source connectors for REST APIs by modifying boilerplate YAML files.
 
 ### Python Connector-Development Kit \(CDK\)
 

docs/connector-development/config-based/low-code-cdk-overview.md

@@ -1,6 +1,6 @@
 # Low-code connector development
 
-Airbyte’s low-code framework enables you to build source connectors for REST APIs via a [connector builder UI](https://docs.airbyte.com/connector-development/config-based/connector-builder-ui) or by modifying boilerplate YAML files via terminal or text editor.
+Airbyte’s low-code framework enables you to build source connectors for REST APIs via a [connector builder UI](https://docs.airbyte.com/connector-development/connector-builder-ui/overview) or by modifying boilerplate YAML files via terminal or text editor.
 
 :::info 
 Developer updates will be announced via our #using-the-cdk Slack channel. If you are using the CDK, please join to stay up to date on changes and issues. 
@@ -83,7 +83,7 @@ For a step-by-step tutorial, refer to the [Getting Started tutorial](./tutorial/
 ## Connector Builder UI
 The main concept powering the lowcode connector framework is the Connector Manifest, a YAML file which describes the features and functionality of the connector. The structure of this YAML file is described in more detail [here](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/yaml-overview). 
 
-We recommend iterating on this YAML file is via the [connector builder UI](https://docs.airbyte.com/connector-development/config-based/connector-builder-ui) as it makes it easy to inspect and debug your connector in greater detail than you would be able to through the commandline. While you can still iterate via the commandline (and the docs contain instructions for how to do it), we're investing heavily in making the UI give you iteration superpowers, so we recommend you check it out!
+We recommend iterating on this YAML file is via the [connector builder UI](https://docs.airbyte.com/connector-development/connector-builder-ui/overview) as it makes it easy to inspect and debug your connector in greater detail than you would be able to through the commandline. While you can still iterate via the commandline (and the docs contain instructions for how to do it), we're investing heavily in making the UI give you iteration superpowers, so we recommend you check it out!
 
 ## Configuring the YAML file
 

docs/connector-development/connector-builder-ui/authentication.md

@@ -91,7 +91,7 @@ Depending on how the refresh endpoint is implemented exactly, additional configu
 * **Token expire property date format** - if not specified, the expiry property is interpreted as the number of seconds the access token will be valid
 * **Access token property name** - the name of the property in the response that contains the access token to do requests. If not specified, it's set to `access_token`
 
-If the API uses a short-lived refresh token that expires after a short amount of time and needs to be refreshed as well or if other grant types like PKCE are required, it's not possible to use the connector builder with OAuth authentication - check out the [compatibility guide](/connector-development/config-based/connector-builder-compatibility#oauth) for more information.
+If the API uses a short-lived refresh token that expires after a short amount of time and needs to be refreshed as well or if other grant types like PKCE are required, it's not possible to use the connector builder with OAuth authentication - check out the [compatibility guide](/connector-development/connector-builder-ui/connector-builder-compatibility#oauth) for more information.
 
 Keep in mind that the OAuth authentication method does not implement a single-click authentication experience for the end user configuring the connector - it will still be necessary to obtain client id, client secret and refresh token from the API and manually enter them into the configuration form.
 

docs/connector-development/config-based/connector-builder-compatibility.md → docs/connector-development/connector-builder-ui/connector-builder-compatibility.md

@@ -1,4 +1,4 @@
-# Connector Builder Compatibility Guide
+# Compatibility Guide
 Answer the following questions to determine whether the Connector Builder is the right tool to build the connector you need:
 - [ ] [Is it an HTTP API returning a collection of records synchronously?](#is-the-integration-an-http-api-returning-a-collection-of-records-synchronously)
 - [ ] [Is the API using one of the following authentication mechanism?](#what-type-of-authentication-is-required)

docs/connector-development/connector-builder-ui/error-handling.md

@@ -111,4 +111,4 @@ The "Error message" field is used to customize the message that is relayed back
 
 ## Multiple error handlers
 
-In the "Error handlers" section of a stream, one or more handlers can be defined. In the case multiple error handlers are specified, the response will be evaluated against each error handler in the order they are defined. The connector will take the action of the first error handler that matches the response and ignore subsequent handlers.
+In the "Error handlers" section of a stream, one or more handlers can be defined. In the case multiple error handlers are specified, the response will be evaluated against each error handler in the order they are defined. The connector will take the action of the first error handler that matches the response and ignore subsequent handlers.

docs/connector-development/connector-builder-ui/overview.md

@@ -0,0 +1,75 @@
+# Connector Builder Intro
+
+The connector builder UI provides an ergonomic iteration interface on top of the [low-code YAML format](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/yaml-overview) and to use built connectors for syncs within the same workspace directly from within the UI. We recommend using it to iterate on your low-code connectors.
+
+:::caution
+The connector builder UI is in beta, which means it’s still in active development and may include backward-incompatible changes. Share feedback and requests with us on our Slack channel or email us at feedback@airbyte.io
+
+**The connector builder does not yet have a link in the sidebar, to navigate to it manually change the URL path in your browsers address bar to `/<your workspace id>/connector-builder`**
+
+Developer updates will be announced via our #public-help-connector-development Slack channel. If you are using the CDK, please join to stay up to date on changes and issues.
+:::
+
+
+## When should I use the connector builder?
+
+The connector builder is the right tool if the following points are met:
+* You want to integrate with a JSON-based HTTP API as a source of records
+* The API you want to integrate with doesn't exist yet as a connector in the [connector catalog](./category/sources).
+* The API is suitable for the connector builder as per the
+[compatibility guide](./connector-builder-compatibility.md).
+
+## Getting started
+
+The high level flow for using the connector builder is as follows:
+
+1. Run the Airbyte webapp to access the connector builder
+2. Use the connector builder to iterate on your low-code connector
+3. Once the connector is ready, publish it to the local workspace
+4. Configure a Source based on the released connector
+5. Use the Source in a connection to sync data
+
+Follow [the tutorial](./tutorial.mdx) for an example of this flow. The concept pages in the side bar to the left go into greater detail of more complex configurations.
+
+## Connector vs. configured source vs. connection
+
+When building a connector, it's important to differentiate between the connector, the configured source based on a connector and the connection:
+
+The **connector** defines the functionality how to access an API or a database, for example protocol, URL paths to access, the way requests need to be structured and how to extract records from responses.
+
+:::info
+While configuring a connector in the builder, make sure to not hardcode things like API keys or passwords - these should be passed in as user input when configuring a Source based on your connector.
+
+Follow [the tutorial](./tutorial.mdx) for an example how this looks like in practice.
+:::
+
+The **configured source** is configuring a connector to actually extract records. The exact fields of the configuration depend on the connector, but in most cases it provides authentication information (username and password, api key) and information about which data to extract, for example start date to sync records from, a search query records have to match.
+
+The **connection** links up a configured source and a configured destination to perform syncs. It defines things like the replication frequency (e.g. hourly, daily, manually) and which streams to replicate.
+
+## Exporting the connector
+
+:::info
+This section is only relevant if you want to contribute your connector back to the Airbyte connector catalog to make it available outside of your workspace.
+:::
+
+The connector builder leverages the [low-code CDK](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/yaml-overview) under the hood, turning all configurations into the YAML format. While in most cases it's not necessary to interact with the YAML representation, it can be used to export the connector specification into a file and build a docker image containing the connector which can be shared more widely:
+
+2. Use the connector builder to iterate on your low-code connector
+3. Export the YAML into a low-code connector module on your local machine
+4. Build the connector's Docker image
+5. Use the built connector image in Airbyte
+
+Once you're done iterating on your connector in the UI, you'll need to export the low-code YAML representation of the connector to your local filesystem into a connector module. This YAML can be downloaded by clicking the `Download Config` button in the bottom-left.
+
+Create a low-code connector module using the connector generator (see [this YAML tutorial for an example](../config-based/tutorial/1-create-source.md)) using the name you'd like to use for your connector. After creating the connector, overwrite the contents of `airbyte-integrations/connectors/source-<connector name>/source_<connector name>/manifest.yaml` with the YAML you created in the UI.
+
+Follow the instructions in the connector README to build the Docker image. Typically this will be something like `docker build . -t airbyte/source-<name>:<version>`.
+
+From this point on your connector is a regular low-code CDK connector and can be distributed as a docker image and made part of the regular Airbyte connector catalog - you can find the [publish process on the overview page](/connector-development/#publishing-a-connector).
+
+### Building the connector image
+
+Follow the instructions in the connector README to build the Docker image. Typically this will be something like `docker build . -t airbyte/source-<name>:<version>`.
+
+Once you've built the connector image, [follow these instructions](https://docs.airbyte.com/integrations/custom-connectors#adding-your-connectors-in-the-ui) to add your connector to your Airbyte instance.

docs/connector-development/connector-builder-ui/record-processing.mdx

@@ -374,4 +374,4 @@ The `subject` field was detected as `string`, but is configured to be a `number`
 * If the API changed and subject is always a string now, the declared schema should be updated to reflect this: `"subject": { "type": "string" }`
 * If the API is sometimes returning subject as number of string depending on the record, the declared schema should be updated to allow both data types: `"subject": { "type": ["string","number"] }`
 
-A common situation is that certain record fields do not have any any values for the test read data, so they are set to `null`. In the detected schema, these field are of type `"null"` which is most likely not correct for all cases. In these situations, the declared schema should be manually corrected.
+A common situation is that certain record fields do not have any any values for the test read data, so they are set to `null`. In the detected schema, these field are of type `"null"` which is most likely not correct for all cases. In these situations, the declared schema should be manually corrected.