Connector builder server: Add inferred schema to read API response #20942
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
January 2, 2023 09:58
added 2 commits
January 6, 2023 13:48
octavia-squidington-iv
added
CDK
added 2 commits
January 9, 2023 11:48
…airbyte into flash1293/schema-inferrer-api
|
Unit tests are not working right currently due to unrelated changes, this is fixed on #21107 |
girarda
requested changes
Jan 9, 2023
airbyte-connector-builder-server/connector_builder/impl/default_api.py
Outdated
Show resolved
Hide resolved
| @@ -108,12 +111,14 @@ async def read_stream(self, stream_read_request_body: StreamReadRequestBody = Bo | |||
| :return: Airbyte record messages produced by the sync grouped by slice and page | |||
| """ | |||
| adapter = self._create_low_code_adapter(manifest=stream_read_request_body.manifest) | |||
| schema_inferrer = SchemaInferrer() | |||
There was a problem hiding this comment.
this class is stateful right? we'll need to reset it or create a new SchemaInferrer for every read to avoid leaking the type of records produced from a different configuration
There was a problem hiding this comment.
isn't this a local variable in the scope of the current read_stream invocation? If that's the case there should be a new schema inferrer for every call of read_stream which is exactly what we want, right?
girarda
approved these changes
Jan 9, 2023
| @@ -108,12 +111,14 @@ async def read_stream(self, stream_read_request_body: StreamReadRequestBody = Bo | |||
| :return: Airbyte record messages produced by the sync grouped by slice and page | |||
| """ | |||
| adapter = self._create_low_code_adapter(manifest=stream_read_request_body.manifest) | |||
| schema_inferrer = SchemaInferrer() | |||
jbfbell
pushed a commit
that referenced
this pull request
Jan 13, 2023
…20942) * fix stuff * add inferred schema to API * fix yaml changes * fix yaml formatting * add whitespace back * reorder imports
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #20831
What
Adds the inferred schema for all encountered records to the response of the read API. Will be consumed by the UI in a subsequent PR to allow to add the schema for a stream with a single click
How
Uses the utility class introduced in #20941 to run all records through the inferrer and attach the final response to the response.