🐞 Destination S3 & GCS Avro: support array with unknown item type #9367

tuliren · 2022-01-09T19:38:07Z

What

When the array field has an items property, but the property is empty (i.e. without a type specification), the Json to Avro schema converter will throw exception.
This PR defaults such array to an array of strings.
This PR also defaults any kind of exception to the string schema. We should not see errors from the Json to Avro schema converter any more.
Resolve 🐛 Source Slack: more specific typing required for pending_shared field #9252.

🚨 User Impact 🚨

This PR should not affect any existing connections.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

tuliren · 2022-01-09T20:01:35Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1674755083
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1674755083
No Python unittests run

tuliren · 2022-01-09T20:01:40Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1674755215
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1674755215
No Python unittests run

tuliren · 2022-01-09T20:01:45Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1674756397
❌ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1674756397
🐛

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1674756397
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1674756397
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%

tuliren · 2022-01-09T20:01:48Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1674756187
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1674756187
🐛

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1674756187
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1674756187
🐛

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1674756187
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1674756187
🐛

tuliren · 2022-01-10T00:47:34Z

Integration test failed due to Could not find image: airbyte/normalization:dev. I wonder if this is due to insufficient disk space of our Github runner.

edgao

Should the json/avro conversion doc also mention that any errors will default to string-type? (everything else seems fine!)

docs/integrations/destinations/bigquery.md

alafanechere

LGTM, thank you for the fix!

tuliren · 2022-01-12T06:43:16Z

/publish connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1686288639
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1686288639

tuliren · 2022-01-12T06:43:20Z

/publish connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1686288720
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1686288720

tuliren · 2022-01-12T07:09:43Z

airbyte-config/init/src/main/resources/seed/destination_specs.yaml

@@ -3594,8 +3594,6 @@
                  \ more memory. Allowed values: min=5MB, max=525MB Default: 5MB."
                type: "integer"
                default: 5
-                minimum: 5
-                maximum: 525


I am really confused why the spec changed. I did not touch the spec.json at all. Also it does not seem that any recent PR has removed these fields. Actually these fields should not exist in the first place.

Support array field with empty items specification

a8ceadc

github-actions bot added the area/connectors Connector related issues label Jan 9, 2022

tuliren changed the title ~~Support array field with empty items specification~~ 🐞 Destination S3 & GCS: support array with unknown item type Jan 9, 2022

Remove all exceptions

da3d3b8

tuliren requested review from alafanechere and edgao January 9, 2022 19:49

tuliren temporarily deployed to more-secrets January 9, 2022 19:51 Inactive

tuliren added 2 commits January 9, 2022 11:55

Format code

c3c44ee

Bump connector versions

eb11020

tuliren temporarily deployed to more-secrets January 9, 2022 19:57 Inactive

tuliren added 2 commits January 9, 2022 11:59

Bump bigquery versions

c2f1145

Update docs

2865f78

github-actions bot added the area/documentation Improvements or additions to documentation label Jan 9, 2022

tuliren temporarily deployed to more-secrets January 9, 2022 20:01 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 9, 2022 20:03 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 9, 2022 20:04 Inactive

Remove unused code

224ad2b

tuliren temporarily deployed to more-secrets January 9, 2022 20:09 Inactive

Update doc for PR #9363

c0148c3

tuliren mentioned this pull request Jan 10, 2022

Refactor and clean up json avro schema converter #9363

Merged

tuliren temporarily deployed to more-secrets January 10, 2022 00:01 Inactive

edgao approved these changes Jan 10, 2022

View reviewed changes

docs/integrations/destinations/bigquery.md Outdated Show resolved Hide resolved

Update doc about defaulting all improperly typed fields to string

31f06a1

tuliren temporarily deployed to more-secrets January 10, 2022 04:06 Inactive

alafanechere approved these changes Jan 11, 2022

View reviewed changes

Merge branch 'master' into liren/support-json-items-without-type

b6aa23d

tuliren temporarily deployed to more-secrets January 12, 2022 06:37 Inactive

tuliren added 2 commits January 11, 2022 22:41

Ignore bigquery

244f787

Update version and doc

56eb4ab

tuliren changed the title ~~🐞 Destination S3 & GCS: support array with unknown item type~~ 🐞 Destination S3 & GCS Avro: support array with unknown item type Jan 12, 2022

tuliren temporarily deployed to more-secrets January 12, 2022 06:44 Inactive

Update doc

d57c560

tuliren temporarily deployed to more-secrets January 12, 2022 06:46 Inactive

Bump version in seed

35d300a

tuliren temporarily deployed to more-secrets January 12, 2022 07:09 Inactive

tuliren commented Jan 12, 2022

View reviewed changes

tuliren merged commit 5f6785d into master Jan 12, 2022

tuliren deleted the liren/support-json-items-without-type branch January 12, 2022 07:12

This was referenced Jan 13, 2022

Bump Airbyte version from 0.35.5-alpha to 0.35.6-alpha #9483

Closed

Bump Airbyte version from 0.35.5-alpha to 0.35.6-alpha #9630

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐞 Destination S3 & GCS Avro: support array with unknown item type #9367

🐞 Destination S3 & GCS Avro: support array with unknown item type #9367

tuliren commented Jan 9, 2022 •

edited

Loading

tuliren commented Jan 9, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 9, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 9, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 9, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 10, 2022

edgao left a comment

alafanechere left a comment

tuliren commented Jan 12, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 12, 2022 •

edited by github-actions bot

Loading

tuliren Jan 12, 2022 •

edited

Loading

🐞 Destination S3 & GCS Avro: support array with unknown item type #9367

🐞 Destination S3 & GCS Avro: support array with unknown item type #9367

Conversation

tuliren commented Jan 9, 2022 • edited Loading

What

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

tuliren commented Jan 9, 2022 • edited by github-actions bot Loading

tuliren commented Jan 9, 2022 • edited by github-actions bot Loading

tuliren commented Jan 9, 2022 • edited by github-actions bot Loading

tuliren commented Jan 9, 2022 • edited by github-actions bot Loading

tuliren commented Jan 10, 2022

edgao left a comment

Choose a reason for hiding this comment

alafanechere left a comment

Choose a reason for hiding this comment

tuliren commented Jan 12, 2022 • edited by github-actions bot Loading

tuliren commented Jan 12, 2022 • edited by github-actions bot Loading

tuliren Jan 12, 2022 • edited Loading

Choose a reason for hiding this comment

tuliren commented Jan 9, 2022 •

edited

Loading

tuliren commented Jan 9, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 9, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 9, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 9, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 12, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 12, 2022 •

edited by github-actions bot

Loading

tuliren Jan 12, 2022 •

edited

Loading