Update lowcode docs (#17752)

* Update lowcode docs * Update * fix links * more links * Schemas * indent * Update docs/connector-development/config-based/advanced-topics.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/advanced-topics.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/low-code-cdk-overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/request-options.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/requester.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/stream-slicers.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/requester.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/error-handling.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * capitalization * fix schema * fix schema * small changes so greenhouse validates * fix brackets * update code samples * Add missing types * fix * reset Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
airbytehq · Oct 8, 2022 · 39220f4 · 39220f4
1 parent 7009bbe
commit 39220f4
Show file tree

Hide file tree

Showing 13 changed files with 1,575 additions and 192 deletions.
diff --git a/...python/airbyte_cdk/sources/declarative/requesters/error_handlers/default_error_handler.py b/...python/airbyte_cdk/sources/declarative/requesters/error_handlers/default_error_handler.py
@@ -54,8 +54,9 @@ class DefaultErrorHandler(ErrorHandler, JsonSchemaMixin):
     4. ignore HTTP 404
     `
       error_handler:
-        - http_codes: [ 404 ]
-          action: IGNORE
+        response_filters:
+          - http_codes: [ 404 ]
+            action: IGNORE
     `
     5. retry if error message contains `retrythisrequest!` substring
     `

diff --git a/docs/connector-development/config-based/advanced-topics.md b/docs/connector-development/config-based/advanced-topics.md
@@ -65,11 +65,21 @@ TopLevel(param=ParamType(k="v"))
 
 More details on object instantiation can be found [here](https://airbyte-cdk.readthedocs.io/en/latest/api/airbyte_cdk.sources.declarative.parsers.html?highlight=factory#airbyte_cdk.sources.declarative.parsers.factory.DeclarativeComponentFactory).
 
-### $options
+## $options
 
 Parameters can be passed down from a parent component to its subcomponents using the $options key.
 This can be used to avoid repetitions.
 
+Schema:
+
+```yaml
+  "$options":
+    type: object
+    additionalProperties: true
+```
+
+Example:
+
 ```yaml
 outer:
   $options:
@@ -230,7 +240,7 @@ some_object:
 ```
 
 Some components also pass in additional arguments to the context.
-This is the case for the [record selector](understanding-the-yaml-file/record-selector.md), which passes in an additional `response` argument.
+This is the case for the [record selector](./understanding-the-yaml-file/record-selector.md), which passes in an additional `response` argument.
 
 Both dot notation and bracket notations (with single quotes ( `'`)) are interchangeable.
 This means that both these string templates will evaluate to the same string:
@@ -244,11 +254,11 @@ For example,
 
 The macros available can be found [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/macros.py).
 
-Additional information on jinja templating can be found at https://jinja.palletsprojects.com/en/3.1.x/templates/#
+Additional information on jinja templating can be found at [https://jinja.palletsprojects.com/en/3.1.x/templates/#](https://jinja.palletsprojects.com/en/3.1.x/templates/#)
 
 ## Component schema reference
 
-A JSON schema representation of the relationships between the components that can be used in the YAML configuration can be found [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/airbyte_cdk/sources/declarative/config_component_schema.json).
+A JSON schema representation of the relationships between the components that can be used in the YAML configuration can be found [here](./source_schema.yaml).
 
 ## Custom components
 
@@ -281,4 +291,21 @@ This class can then be referred from the yaml file using its fully qualified cla
 pagination_strategy:
   class_name: "my_connector_module.MyPaginationStrategy"
   my_field: "hello world"
-```
+```
+
+## How the framework works
+
+1. Given the connection config and an optional stream state, the `StreamSlicer` computes the stream slices to read.
+2. Iterate over all the stream slices defined by the stream slicer.
+3. For each stream slice,
+    1. Submit a request to the partner API as defined by the requester
+    2. Select the records from the response
+    3. Repeat for as long as the paginator points to a next page
+
+[connector-flow](./assets/connector-flow.png)
+
+## More readings
+
+- [Record selector](./understanding-the-yaml-file/record-selector.md)
+- [Stream slicers](./understanding-the-yaml-file/stream-slicers.md)
+- [Source schema](./source_schema.yaml)
diff --git a/docs/connector-development/config-based/assets/connector-flow.png b/docs/connector-development/config-based/assets/connector-flow.png
diff --git a/docs/connector-development/config-based/low-code-cdk-overview.md b/docs/connector-development/config-based/low-code-cdk-overview.md
@@ -2,7 +2,7 @@
 
 Airbyte’s low-code framework enables you to build source connectors for REST APIs by modifying boilerplate YAML files.
 
-:::warning 
+:::warning
 The low-code framework is in [alpha](https://docs.airbyte.com/project-overview/product-release-stages/#alpha), which means it’s still in active development and may include backward-incompatible changes. Share feedback and requests with us on our [Slack channel](https://slack.airbyte.com/) or email us at [feedback@airbyte.io](mailto:feedback@airbyte.io)
 :::
 
@@ -11,29 +11,29 @@ The low-code framework is in [alpha](https://docs.airbyte.com/project-overview/p
 Refer to the REST API documentation for the source you want to build the connector for and answer the following questions:
 
 - Does the REST API documentation show which HTTP method to use to retrieve data, and that the response is a JSON object?
-- Do the queries either return data synchronously or trigger a bulk workflow?
+- Do the queries return data synchronously?
 - Does the API support any of the following pagination mechanisms:
-    - Offset count passed either by query params or request header 
-    - Page count passed either by query params or request header 
-    - Cursor field pointing to the URL of the next page of records 
+    - Offset count passed either by query params or request header
+    - Page count passed either by query params or request header
+    - Cursor field pointing to the URL of the next page of records
 - Does the API support any of the following authentication mechanisms:
     - [A query param or a HTTP header](https://docs.airbyte.com/connector-development/config-based/authentication#apikeyauthenticator)
     - [Basic Auth over HTTPS](https://docs.airbyte.com/connector-development/config-based/authentication#basichttpauthenticator)
     - [OAuth 2.0](https://docs.airbyte.com/connector-development/config-based/authentication#oauth)
 - Does the API support static schema?
 - Does the endpoint have a strict rate limit?
-    Throttling is not supported, but the connector can use exponential backoff to avoid API bans in case it gets rate limited. This can work for APIs with high rate limits, but not for those that have strict limits on a small time-window.
+  Throttling is not supported, but the connector can use exponential backoff to avoid API bans in case it gets rate limited. This can work for APIs with high rate limits, but not for those that have strict limits on a small time-window.
 - Are the following features sufficient:
 
-    | Feature                                                      | Support                                                                                                                                                                                                                                       |
-    |--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-    | Resource type                                                | Collections<br/>[Sub-collection](understanding-the-yaml-file/stream-slicers.md#substreams)                                                                                                                                                      |
-    | Sync mode                                                    | Full refresh<br/>Incremental                                                                                                                                             |
-    | Schema discovery                                             | Static schemas                                                                                                                                                  |
-    | [Stream slicing](understanding-the-yaml-file/stream-slicers.md)                        | [Datetime](understanding-the-yaml-file/stream-slicers.md#Datetime), [lists](understanding-the-yaml-file/stream-slicers.md#list-stream-slicer), [parent-resource id](understanding-the-yaml-file/stream-slicers.md#Substream-slicer)                                                                                                                                     |
-    | [Record transformation](understanding-the-yaml-file/record-selector.md)                | [Field selection](understanding-the-yaml-file/record-selector.md#selecting-a-field)<br/>[Adding fields](understanding-the-yaml-file/record-selector.md#adding-fields)<br/>[Removing fields](understanding-the-yaml-file/record-selector.md#removing-fields)<br/>[Filtering records](understanding-the-yaml-file/record-selector.md#filtering-records) |
-    | [Error detection](understanding-the-yaml-file/error-handling.md)                       | [From HTTP status  code](understanding-the-yaml-file/error-handling.md#from-status-code)<br/>[From error message](understanding-the-yaml-file/error-handling.md#from-error-message)                                                                                                               |
-    | [Backoff strategies](understanding-the-yaml-file/error-handling.md#Backoff-Strategies) | [Exponential](understanding-the-yaml-file/error-handling.md#Exponential-backoff)<br/>[Constant](understanding-the-yaml-file/error-handling.md#Constant-Backoff)<br/>[Derived from headers](understanding-the-yaml-file/error-handling.md#Wait-time-defined-in-header)                                                       |
+  | Feature                                                      | Support                                                                                                                                                                                                                                       |
+              |--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+  | Resource type                                                | Collections<br/>[Sub-collection](understanding-the-yaml-file/stream-slicers.md#substreams)                                                                                                                                                      |
+  | Sync mode                                                    | Full refresh<br/>Incremental                                                                                                                                             |
+  | Schema discovery                                             | Static schemas                                                                                                                                                  |
+  | [Stream slicing](understanding-the-yaml-file/stream-slicers.md)                        | [Datetime](understanding-the-yaml-file/stream-slicers.md#Datetime), [lists](understanding-the-yaml-file/stream-slicers.md#list-stream-slicer), [parent-resource id](understanding-the-yaml-file/stream-slicers.md#Substream-slicer)                                                                                                                                     |
+  | [Record transformation](understanding-the-yaml-file/record-selector.md)                | [Field selection](understanding-the-yaml-file/record-selector.md#selecting-a-field)<br/>[Adding fields](understanding-the-yaml-file/record-selector.md#adding-fields)<br/>[Removing fields](understanding-the-yaml-file/record-selector.md#removing-fields)<br/>[Filtering records](understanding-the-yaml-file/record-selector.md#filtering-records) |
+  | [Error detection](understanding-the-yaml-file/error-handling.md)                       | [From HTTP status  code](understanding-the-yaml-file/error-handling.md#from-status-code)<br/>[From error message](understanding-the-yaml-file/error-handling.md#from-error-message)                                                                                                               |
+  | [Backoff strategies](understanding-the-yaml-file/error-handling.md#Backoff-Strategies) | [Exponential](understanding-the-yaml-file/error-handling.md#Exponential-backoff)<br/>[Constant](understanding-the-yaml-file/error-handling.md#Constant-Backoff)<br/>[Derived from headers](understanding-the-yaml-file/error-handling.md#Wait-time-defined-in-header)                                                       |
 
 If the answer to all questions is yes, you can use the low-code framework to build a connector for the source. If not, use the [Python CDK](../cdk-python/README.md).
 
@@ -101,7 +101,19 @@ For each stream, configure the following components:
 | Transformations        |                 | A set of transformations to be applied on the records read from the source before emitting them to the destination                                                                                                                    |
 | Checkpoint interval    |                 | Defines the interval, in number of records, at which incremental syncs should be checkpointed                                                                                                                                         |
 
-For a deep dive into each of the components, refer to Understanding the YAML file
+For a deep dive into each of the components, refer to [Understanding the YAML file](./understanding-the-yaml-file/yaml-overview.md) or the [full YAML Schema definition](./source_schema.yaml)
+
+## Tutorial
+
+This section a tutorial that will guide you through the end-to-end process of implementing a low-code connector.
+
+0. [Getting started](tutorial/0-getting-started.md)
+1. [Creating a source](tutorial/1-create-source.md)
+2. [Installing dependencies](tutorial/2-install-dependencies.md)
+3. [Connecting to the API](tutorial/3-connecting-to-the-API-source.md)
+4. [Reading data](tutorial/4-reading-data.md)
+5. [Incremental reads](tutorial/5-incremental-reads.md)
+6. [Testing](tutorial/6-testing.md)
 
 ## Sample connectors