Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

File-based CDK: allow null values for all inferred columns #28847

Merged
merged 2 commits into from
Jul 31, 2023
Merged

File-based CDK: allow null values for all inferred columns #28847

merged 2 commits into from
Jul 31, 2023

Conversation

clnoll
Copy link
Contributor

@clnoll clnoll commented Jul 31, 2023

What

Allows null values for all columns returned in inferred schemas.

This is consistent with the schema inference in source_files_abstract.

@clnoll clnoll requested a review from a team as a code owner July 31, 2023 02:06
@clnoll clnoll requested a review from brianjlai July 31, 2023 02:06
@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label Jul 31, 2023
@clnoll clnoll force-pushed the file-cdk-schema-inference-allow-null-types branch from e0d4535 to ec25875 Compare July 31, 2023 03:13
@clnoll clnoll requested a review from girarda July 31, 2023 16:16
girarda
girarda approved these changes Jul 31, 2023
View reviewed changes
Copy link
Contributor

@girarda girarda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is great @clnoll ! :shipit:

airbyte-cdk/python/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py
return self._fill_nulls(schema)

@staticmethod
def _fill_nulls(schema: Mapping[str, Any]) -> Mapping[str, Any]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it might be worth unit testing this method instead of only relying on the scenarios since it's not trivial

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, added.

@clnoll clnoll merged commit 73395a1 into master Jul 31, 2023
@clnoll clnoll deleted the file-cdk-schema-inference-allow-null-types branch July 31, 2023 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@girarda girarda girarda approved these changes

@brianjlai brianjlai Awaiting requested review from brianjlai

Assignees
No one assigned
Labels
CDK Connector Development Kit
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@clnoll @girarda @octavia-squidington-iii