Update lowcode docs #17752

girarda · 2022-10-07T17:08:39Z

What

Update low-code documentation as per feedback

How

Fix a code sample in airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/default_error_handler.py
Add YAML schema for all components in the docs
Add complete YAML schema for the YAML file
Add How the framework works section with a flow diagram in advanced topics
Add some more links
Reword a few sentences

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

brianjlai

Docs changes look good! Just a few small suggestions and edits. But my main concern about this is how we keep all the schema blocks up to date as the language changes. This should be okay since we won't modify it much for Hacktoberfest, but this might start to get unwieldly when we try to make future breaking changes.

brianjlai · 2022-10-07T22:43:10Z

docs/connector-development/config-based/source_schema.yaml

@@ -0,0 +1,648 @@
+---


Out of curiosity was this YAML handwritten or generated from the code and if so how?

handwritten

it's not a perfect test, but I validated it against greenhouse, sentry, and sendgrid

docs/connector-development/config-based/advanced-topics.md

docs/connector-development/config-based/low-code-cdk-overview.md

docs/connector-development/config-based/understanding-the-yaml-file/error-handling.md

docs/connector-development/config-based/understanding-the-yaml-file/request-options.md

docs/connector-development/config-based/understanding-the-yaml-file/requester.md

docs/connector-development/config-based/understanding-the-yaml-file/stream-slicers.md

brianjlai · 2022-10-07T23:25:15Z

docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md

+1. [Requester](./requester.md): Describes how to submit requests to the API source
+2. [Paginator](./pagination.md): Describes how to navigate through the API's pages
+3. [Record selector](./record-selector.md): Describes how to extract records from a HTTP response
+4. [Stream Slicer](./stream-slicers.md): Describes how to partition the stream, enabling incremental syncs and checkpointing


lowercase or the previous one uppercase? capitalization is a little inconsistent

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

…-file/request-options.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

…-file/requester.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

…-file/stream-slicers.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

…-file/requester.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

…-file/error-handling.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

girarda · 2022-10-07T17:08:57Z

...dk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/default_error_handler.py

@@ -54,8 +54,9 @@ class DefaultErrorHandler(ErrorHandler, JsonSchemaMixin):
    4. ignore HTTP 404
    `
      error_handler:
-        - http_codes: [ 404 ]
-          action: IGNORE
+        response_filters:


this was a bug in the comment

girarda · 2022-10-07T17:09:52Z

docs/connector-development/config-based/low-code-cdk-overview.md

@@ -11,29 +11,29 @@ The low-code framework is in [alpha](https://docs.airbyte.com/project-overview/p
 Refer to the REST API documentation for the source you want to build the connector for and answer the following questions:

 - Does the REST API documentation show which HTTP method to use to retrieve data, and that the response is a JSON object?
- Do the queries either return data synchronously or trigger a bulk workflow?
+- Do the queries either return data synchronously?


we don't support bulk workflows

girarda · 2022-10-07T17:10:06Z

docs/connector-development/config-based/low-code-cdk-overview.md

-    | [Record transformation](understanding-the-yaml-file/record-selector.md)                | [Field selection](understanding-the-yaml-file/record-selector.md#selecting-a-field)<br/>[Adding fields](understanding-the-yaml-file/record-selector.md#adding-fields)<br/>[Removing fields](understanding-the-yaml-file/record-selector.md#removing-fields)<br/>[Filtering records](understanding-the-yaml-file/record-selector.md#filtering-records) |
-    | [Error detection](understanding-the-yaml-file/error-handling.md)                       | [From HTTP status  code](understanding-the-yaml-file/error-handling.md#from-status-code)<br/>[From error message](understanding-the-yaml-file/error-handling.md#from-error-message)                                                                                                               |
-    | [Backoff strategies](understanding-the-yaml-file/error-handling.md#Backoff-Strategies) | [Exponential](understanding-the-yaml-file/error-handling.md#Exponential-backoff)<br/>[Constant](understanding-the-yaml-file/error-handling.md#Constant-Backoff)<br/>[Derived from headers](understanding-the-yaml-file/error-handling.md#Wait-time-defined-in-header)                                                       |
+  | Feature                                                      | Support                                                                                                                                                                                                                                       |


this is all just autoformating...

girarda · 2022-10-07T17:10:23Z

docs/connector-development/config-based/low-code-cdk-overview.md

@@ -103,6 +103,18 @@ For each stream, configure the following components:

 For a deep dive into each of the components, refer to Understanding the YAML file

+## Tutorial


overall view and links to the tutorial

girarda · 2022-10-07T17:16:06Z

docs/connector-development/config-based/advanced-topics.md

@@ -65,11 +65,21 @@ TopLevel(param=ParamType(k="v"))

 More details on object instantiation can be found [here](https://airbyte-cdk.readthedocs.io/en/latest/api/airbyte_cdk.sources.declarative.parsers.html?highlight=factory#airbyte_cdk.sources.declarative.parsers.factory.DeclarativeComponentFactory).

-### $options
+## $options


move to header2

girarda · 2022-10-07T17:37:24Z

docs/connector-development/config-based/understanding-the-yaml-file/record-selector.md

-    field_pointer:
-      - "data"
-      - "records"
+    field_pointer: [ "data", "records" ]


change to [] for consistency

girarda · 2022-10-07T17:39:23Z

docs/connector-development/config-based/understanding-the-yaml-file/stream-slicers.md

@@ -118,26 +233,58 @@ the resulting stream slices are
 ]
 ```

-[^1] This is a slight oversimplification. See [update cursor section](#cursor-update) for more details on how the cursor is updated.


moved to the bottom of the file

girarda · 2022-10-07T17:39:41Z

docs/connector-development/config-based/understanding-the-yaml-file/stream-slicers.md

 ```

+## Nested streams


make this more explicit in the hope it helps discoverability

girarda · 2022-10-07T17:40:23Z

docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md

-          - path: [ "start_date" ]
-            value: {{ stream_slice[ 'start_date' ] }}
-```
+## Configuring the cursor field for incremental syncs


Explicitly call out incremental syncs

girarda · 2022-10-08T00:00:16Z

docs/connector-development/config-based/source_schema.yaml

@@ -0,0 +1,648 @@
+---


handwritten

…vation * master: (22 commits) Update full-refresh-append.md (#17784) Update full-refresh-overwrite.md (#17783) Update incremental-append.md (#17785) Update incremental-deduped-history.md (#17786) Update cdc.md (#17787) 🪟 🔧 Ignore classnames during jest snapshot comparison (#17773) feat: replace openjdk with amazoncorretto:17.0.4 on connectors for seсurity compliance (#17511) Start testing buildpulse. (#17712) Add missing types to the registry (#17763) jobs db descriptions (#16543) config db data catalog (#16427) Update lowcode docs (#17752) db migrations to support new webhook operations (#17671) Bump Airbyte version from 0.40.13 to 0.40.14 (#17762) September Release Notes (#17754) Revert "Use java-datadog-tracer-base image (#17625)" (#17759) Add connection migrations for schema changes (#17651) Connection Form Refactor - Part Two (#16748) Improve E2E testing around the Connection Form (#17577) Bump strict encrypt version (#17747) ...

* Update lowcode docs * Update * fix links * more links * Schemas * indent * Update docs/connector-development/config-based/advanced-topics.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/advanced-topics.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/low-code-cdk-overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/request-options.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/requester.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/stream-slicers.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/requester.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/error-handling.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * capitalization * fix schema * fix schema * small changes so greenhouse validates * fix brackets * update code samples * Add missing types * fix * reset Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update lowcode docs

c876e1a

github-actions bot added area/documentation Improvements or additions to documentation CDK Connector Development Kit labels Oct 7, 2022

girarda added 5 commits October 7, 2022 10:15

Update

e355cdf

fix links

9e1017e

more links

fb86d8b

Schemas

4e7e1fc

indent

34da09a

girarda marked this pull request as ready for review October 7, 2022 17:40

girarda requested review from a team and brianjlai October 7, 2022 17:40

girarda mentioned this pull request Oct 7, 2022

Update low-code docs #16897

Closed

37 tasks

brianjlai approved these changes Oct 7, 2022

View reviewed changes

girarda and others added 8 commits October 7, 2022 17:00

Update docs/connector-development/config-based/advanced-topics.md

e7dcb87

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/advanced-topics.md

a45a757

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/low-code-cdk-overview.md

02a413a

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/understanding-the-yaml…

c271f78

…-file/request-options.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/understanding-the-yaml…

ebea939

…-file/requester.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/understanding-the-yaml…

e7e1daf

…-file/stream-slicers.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/understanding-the-yaml…

bb42aa3

…-file/requester.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/understanding-the-yaml…

877b9f6

…-file/error-handling.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

girarda commented Oct 8, 2022

View reviewed changes

girarda added 8 commits October 7, 2022 17:03

capitalization

88be8e8

fix schema

9f01498

fix schema

477d264

small changes so greenhouse validates

4b3c7e0

fix brackets

a9f5c23

update code samples

1a9e1d6

Add missing types

93ac18e

fix

637a9ab

reset

cd9d4b4

girarda merged commit 39220f4 into master Oct 8, 2022

girarda deleted the alex/update_docs branch October 8, 2022 02:15

This was referenced Oct 13, 2022

Bump Airbyte version from 0.40.14 to 0.40.15 #17961

Closed

Bump Airbyte version from 0.40.14 to 0.40.15 #17970

Merged

		@@ -103,6 +103,18 @@ For each stream, configure the following components:

		For a deep dive into each of the components, refer to Understanding the YAML file

		## Tutorial

Update lowcode docs #17752

Update lowcode docs #17752

Uh oh!

Conversation

girarda commented Oct 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

Uh oh!

brianjlai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

girarda commented Oct 7, 2022 •

edited

Loading