Connector builder: support for test read with message grouping per slices #23925

girarda · 2023-03-10T00:48:22Z

What

Implement the test read method

Sample command
python main.py read --config secrets/config.json --catalog secrets/configured_catalog.json

Test read

The test read method will run if the "__command" field is set to "test_read". The source will read data from stream defined in the configured catalog. Only one stream at a time is currently supported.

The test_read command can be configured with the optional "__test_read_config" field in the config
The "__test_read_config" should have the following fields

max_pages_per_slice
max_slices
max_records

sample config for a test read:

{
  "api_key": "",
  "__command": "test_read"
  "__test_read_config": {
    "stream_name": "Accounts",
    "max_pages_per_slice": 5,
    "max_slices": 5,
    "max_records": 10
  },
  "__injected_declarative_manifest": {
}

configured catalog:

{
  "streams": [
    {
      "stream": {
        "name": "Accounts",
        "json_schema": {
          "$schema": "http://json-schema.org/draft-07/schema#",
          "type": "object",
          "properties": {}
        },
        "supported_sync_modes": ["full_refresh"],
        "source_defined_cursor": false
      },
      "sync_mode": "full_refresh",
      "destination_sync_mode": "overwrite"
    }
  ]
}

sample configured catalog for resolving the manifest:

{
  "streams": [
    {
      "stream": {
        "name": "resolve_manifest",
        "json_schema": {
          "$schema": "http://json-schema.org/draft-07/schema#",
          "type": "object",
          "properties": {}
        },
        "supported_sync_modes": ["full_refresh"],
        "source_defined_cursor": false
      },
      "sync_mode": "full_refresh",
      "destination_sync_mode": "overwrite"
    }
  ]
}

How

Interface change

Implement the read_stream method, which will be called if the config's command == "test_read"

Test read

Copy the models from the connector builder server and convert them to dataclasses
Copy the tests from
https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-connector-builder-server/unit_tests/connector_builder/impl/test_default_api.py
Copy the bulk of the default_api logic to a new MessageGrouper class
Implement connector_builder.connector_builder_handler.read_stream
Call read_stream from the main

Recommended reading order

airbyte-cdk/python/source_declarative_manifest/main.py
airbyte-cdk/python/connector_builder/connector_builder_handler.py
airbyte-cdk/python/connector_builder/message_grouper.py
airbyte-cdk/python/connector_builder/models.py
airbyte-cdk/python/airbyte_cdk/sources/source.py
airbyte-cdk/python/unit_tests/connector_builder/test_message_grouper.py
airbyte-cdk/python/airbyte_cdk/sources/source.py
airbyte-cdk/python/unit_tests/connector_builder/utils.py

… Builder. Also implements `resolve_manifest` handler

girarda · 2023-03-10T00:50:03Z

airbyte-cdk/python/unit_tests/connector_builder/test_read.py

@@ -0,0 +1,503 @@
+#


I ported these tests from https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-connector-builder-server/unit_tests/connector_builder/impl/test_default_api.py

girarda · 2023-03-10T01:54:22Z

airbyte-cdk/python/connector_builder/connector_builder_handler.py

+def _emitted_at():
+    return int(datetime.now().timestamp()) * 1000
+
+def _create_configure_catalog(stream_name: str) -> ConfiguredAirbyteCatalog:


creating the catalog on the Java side would allow us to keep the same read interface as connectors

girarda · 2023-03-14T19:03:29Z

airbyte-cdk/python/connector_builder/main.py



 def create_source(config: Mapping[str, Any]) -> ManifestDeclarativeSource:
    manifest = config.get("__injected_declarative_manifest")
-    return ManifestDeclarativeSource(manifest)
+    return ManifestDeclarativeSource(manifest, True)


set debug to True so the source returns raw requests and responses

…ex/test_read

clnoll

Looks great @girarda! A couple of questions but no blockers.

clnoll · 2023-03-15T15:18:39Z

airbyte-cdk/python/connector_builder/connector_builder_handler.py

+
+def read_stream(source: DeclarativeSource, config: Mapping[str, Any], configured_catalog: ConfiguredAirbyteCatalog) -> AirbyteMessage:
+    try:
+        if "__test_read_config" not in config:


nit: Feels like this should be optional since we're not requiring any of the keys to be present, and are handling the case where it isn't set.

agreed. fixed

clnoll · 2023-03-15T15:23:23Z

airbyte-cdk/python/connector_builder/message_grouper.py

+class MessageGrouper:
+    logger = logging.getLogger("airbyte.connector-builder")
+
+    def __init__(self, max_pages_per_slice: int, max_slices: int, max_record_limit: int = 1000):


nit (also I know this may just be copy/pasted): Would it be preferable to enforce the maximums in this class, instead of allowing them to be fully configurable?

the main reason why I didn't enforce maximums here is that we'll need to update the value in two places if we decide to increase it. I think the risk of not enforcing a limit here is negligible because we control the caller

…ex/test_read

…ices (airbytehq#23925) * New connector_builder module for handling requests from the Connector Builder. Also implements `resolve_manifest` handler * Automated Commit - Formatting Changes * Rename ConnectorBuilderSource to ConnectorBuilderHandler * Update source_declarative_manifest README * Reorganize * read records * paste unit tests from connector builder server * compiles but tests fail * first test passes * Second test passes * 3rd test passes * one more test * another test * one more test * test * return StreamRead * test * test * rename * test * test * test * main seems to work * Update * Update * Update * Update * update * error message * rename * update * Update * CR improvements * fix test_source_declarative_manifest * fix tests * Update * Update * Update * Update * rename * rename * rename * format * Give connector_builder its own main.py * Update * reset * delete dead code * remove debug print * update test * Update * set right stream * Add --catalog argument * Remove unneeded preparse * Update README * handle error * tests pass * more explicit test * reset * format * fix merge * raise exception * fix * black format * raise with config * update * fix flake * __test_read_config is optional * fix * Automated Commit - Formatting Changes * fix * exclude_unset --------- Co-authored-by: Catherine Noll <noll.catherine@gmail.com> Co-authored-by: clnoll <clnoll@users.noreply.github.com> Co-authored-by: girarda <girarda@users.noreply.github.com>

…ices (#23925) * New connector_builder module for handling requests from the Connector Builder. Also implements `resolve_manifest` handler * Automated Commit - Formatting Changes * Rename ConnectorBuilderSource to ConnectorBuilderHandler * Update source_declarative_manifest README * Reorganize * read records * paste unit tests from connector builder server * compiles but tests fail * first test passes * Second test passes * 3rd test passes * one more test * another test * one more test * test * return StreamRead * test * test * rename * test * test * test * main seems to work * Update * Update * Update * Update * update * error message * rename * update * Update * CR improvements * fix test_source_declarative_manifest * fix tests * Update * Update * Update * Update * rename * rename * rename * format * Give connector_builder its own main.py * Update * reset * delete dead code * remove debug print * update test * Update * set right stream * Add --catalog argument * Remove unneeded preparse * Update README * handle error * tests pass * more explicit test * reset * format * fix merge * raise exception * fix * black format * raise with config * update * fix flake * __test_read_config is optional * fix * Automated Commit - Formatting Changes * fix * exclude_unset --------- Co-authored-by: Catherine Noll <noll.catherine@gmail.com> Co-authored-by: clnoll <clnoll@users.noreply.github.com> Co-authored-by: girarda <girarda@users.noreply.github.com>

clnoll and others added 22 commits March 9, 2023 13:57

New connector_builder module for handling requests from the Connector…

f829ac5

… Builder. Also implements `resolve_manifest` handler

Automated Commit - Formatting Changes

13a9a14

Rename ConnectorBuilderSource to ConnectorBuilderHandler

1c85330

Update source_declarative_manifest README

0dccb4e

Reorganize

f7a475a

read records

bd71e91

paste unit tests from connector builder server

c6ac119

compiles but tests fail

f949521

first test passes

45741cd

Second test passes

0e7d6f4

3rd test passes

81ff6e9

one more test

14aa8ca

another test

b3764ba

one more test

c6040cc

test

5f0ead1

return StreamRead

1614bad

test

b323578

test

9902dd5

rename

3b3255e

test

6242e3a

test

c8631f1

test

fe1da29

octavia-squidington-iii added the CDK Connector Development Kit label Mar 10, 2023

girarda commented Mar 10, 2023

View reviewed changes

girarda added 4 commits March 9, 2023 17:19

main seems to work

5b0750c

Update

7dd5ada

Update

2f048e9

Update

dd778ed

girarda commented Mar 10, 2023

View reviewed changes

Update

e79adf8

girarda added 5 commits March 14, 2023 11:49

raise exception

e37851e

fix

0b809f9

black format

33022c9

raise with config

cccb968

update

a79b43b

girarda commented Mar 14, 2023

View reviewed changes

Merge branch 'master' into alex/test_read

8ca4d64

girarda marked this pull request as ready for review March 14, 2023 19:04

girarda requested a review from a team as a code owner March 14, 2023 19:04

girarda requested review from clnoll and maxi297 March 14, 2023 19:04

girarda added 3 commits March 14, 2023 12:18

fix flake

4ace71b

Merge branch 'alex/test_read' of github.com:airbytehq/airbyte into al…

e74c040

…ex/test_read

Merge branch 'master' into alex/test_read

fa198be

girarda changed the title ~~Alex/test read~~ Connector builder: support for test read with message grouping per slices Mar 15, 2023

clnoll approved these changes Mar 15, 2023

View reviewed changes

girarda and others added 8 commits March 15, 2023 08:50

__test_read_config is optional

455e65a

Merge branch 'master' into alex/test_read

4059751

merge

f78637c

fix

36152a1

Automated Commit - Formatting Changes

b308f04

fix

40550b3

Merge branch 'alex/test_read' of github.com:airbytehq/airbyte into al…

ae66445

…ex/test_read

exclude_unset

b1beeb3

girarda merged commit bb5741a into master Mar 16, 2023

girarda deleted the alex/test_read branch March 16, 2023 00:12

girarda mentioned this pull request Mar 17, 2023

Connector builder: Port "send stacktrace when error on read" to CDK connector builder module #24173

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connector builder: support for test read with message grouping per slices #23925

Connector builder: support for test read with message grouping per slices #23925

girarda commented Mar 10, 2023 •

edited

Loading

girarda Mar 10, 2023

girarda Mar 10, 2023

girarda Mar 14, 2023

clnoll left a comment

clnoll Mar 15, 2023

girarda Mar 15, 2023

clnoll Mar 15, 2023

girarda Mar 15, 2023

Connector builder: support for test read with message grouping per slices #23925

Connector builder: support for test read with message grouping per slices #23925

Conversation

girarda commented Mar 10, 2023 • edited Loading

What

Test read

How

Interface change

Test read

Recommended reading order

girarda Mar 10, 2023

Choose a reason for hiding this comment

girarda Mar 10, 2023

Choose a reason for hiding this comment

girarda Mar 14, 2023

Choose a reason for hiding this comment

clnoll left a comment

Choose a reason for hiding this comment

clnoll Mar 15, 2023

Choose a reason for hiding this comment

girarda Mar 15, 2023

Choose a reason for hiding this comment

clnoll Mar 15, 2023

Choose a reason for hiding this comment

girarda Mar 15, 2023

Choose a reason for hiding this comment

girarda commented Mar 10, 2023 •

edited

Loading