Avro destinations: map airbyte_type: integer to long #25570

edgao · 2023-04-26T17:05:27Z

What

#23466 only fixed this behavior for {type: integer}. This PR adds the fix for {type: number, airbyte_type: integer}.

Also updated test cases:

add a test case for {type: integer}
update the avroSchema for {type: number, airbyte_type: integer} to be long

(also, submitted an issue to clean up the data types stuff - it's a pretty messy change so will do it separately https://github.com/airbytehq/airbyte-internal-issues/issues/1726)

github-actions · 2023-04-26T17:10:38Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (10)

Connector	Version	Changelog	Publish
`destination-bigquery`	`1.4.0`	✅	✅
`destination-bigquery-denormalized`	`1.4.0`	✅	✅
`destination-databricks`	`1.0.2`	✅	✅
`destination-gcs`	`0.3.0`	✅	✅
`destination-r2`	`0.1.0`	✅	✅
`destination-redshift`	`0.4.7`	✅	✅
`destination-s3`	`0.4.0`	✅	✅
`destination-s3-glue`	`0.1.7`	✅	✅
`destination-snowflake`	`1.0.1`	✅	✅
`destination-starburst-galaxy`	`0.0.1`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

evantahler

Tests pass, so !

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* publish normalization * bump normalization container version in all the destinations that use it Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: edgao <edgao@users.noreply.github.com>

…ess (#25588) * add AIRBYTE_ENTRYPOINT env var for kube * amazing, absolute genius * version bump + changelog * derp, no need to publish denormalizeid * fix changelog entry * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

edgao · 2023-04-28T22:59:17Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/4835432753
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/4835432753
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

edgao · 2023-04-28T22:59:21Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/4835433010
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/4835433010
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

edgao · 2023-04-28T22:59:24Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/4835433278
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/4835433278
No Python unittests run

Build Passed

Test summary info:

All Passed

edgao · 2023-04-28T22:59:27Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/4835433584
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/4835433584
No Python unittests run

Build Passed

Test summary info:

All Passed

edgao · 2023-05-05T13:18:54Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4893854973
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4893854973
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

edgao · 2023-05-05T15:21:34Z

/publish connector=connectors/destination-bigquery,connectors/destination-bigquery-denormalized,connectors/destination-gcs,connectors/destination-s3,connectors/destination-snowflake

🕑 Publishing the following connectors:
connectors/destination-bigquery,connectors/destination-bigquery-denormalized,connectors/destination-gcs,connectors/destination-s3,connectors/destination-snowflake
https://github.com/airbytehq/airbyte/actions/runs/4894973750

Connector	Version	Did it publish?	Were definitions generated?
connectors/destination-gcs	0.3.0	✅	✅
connectors/destination-bigquery-denormalized	1.4.0	✅	✅
connectors/destination-bigquery	1.4.0	✅	✅
connectors/destination-s3	0.4.0	✅	✅
connectors/destination-snowflake	1.0.1	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

tybernstein · 2023-05-10T12:57:07Z

@edgao is there a chance that this would change the datatype format in the following manner:

1676500610 now formatted as 1.678177401E9?

edgao · 2023-05-10T14:45:33Z

I would not expect that - that actually sounds like it's being treated as a number/float. Probably easier to discuss over slack / separate issue though

…#25570) * map the other integer schema to long * fix test + add test * delete_public_access_block for bucket if public (airbytehq#25663) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * 🐛 Source Facebook Marketing: fix `expected records` for CAT (airbytehq#25604) * publish normalization (airbytehq#25591) * publish normalization * bump normalization container version in all the destinations that use it Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: edgao <edgao@users.noreply.github.com> * Bump Airbyte version from 0.44.2 to 0.44.3 * Destination Bigquery: update AIRBYTE_ENTRYPOINT env var for kube process (airbytehq#25588) * add AIRBYTE_ENTRYPOINT env var for kube * amazing, absolute genius * version bump + changelog * derp, no need to publish denormalizeid * fix changelog entry * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * version bumps + changelog * whoops * bump metadata * bump metadatas * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version --------- Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> Co-authored-by: Jeff Cowan (Airbyte) <4992320+jcowanpdx@users.noreply.github.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

map the other integer schema to long

c075b5d

This comment was marked as outdated.

Sign in to view

fix test + add test

6b73086

edgao marked this pull request as ready for review April 26, 2023 17:43

edgao requested a review from a team as a code owner April 26, 2023 17:43

edgao changed the title ~~map the other integer schema to long~~ Destinations S3, Bigquery-denormalized: map airbyte_type: integer to long Apr 26, 2023

This comment was marked as outdated.

Sign in to view

evantahler approved these changes Apr 26, 2023

View reviewed changes

Merge branch 'master' into edgao/dest_s3_ints

e4429ae

This comment was marked as outdated.

Sign in to view

Merge branch 'master' into edgao/dest_s3_ints

b5f59cf

This comment was marked as outdated.

Sign in to view

grubberr and others added 6 commits April 28, 2023 15:54

delete_public_access_block for bucket if public (#25663)

e0ba2d0

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

🐛 Source Facebook Marketing: fix expected records for CAT (#25604)

5f97ebb

publish normalization (#25591)

688c3d5

* publish normalization * bump normalization container version in all the destinations that use it Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: edgao <edgao@users.noreply.github.com>

Bump Airbyte version from 0.44.2 to 0.44.3

86c1d88

version bumps + changelog

81a2259

octavia-squidington-iii added the connectors/source/s3 label Apr 28, 2023

Merge branch 'master' into edgao/dest_s3_ints

341b60c

edgao changed the title ~~Destinations S3, Bigquery-denormalized: map airbyte_type: integer to long~~ Avro destinations: map airbyte_type: integer to long Apr 28, 2023

This comment was marked as outdated.

Sign in to view

Merge branch 'master' into edgao/dest_s3_ints

47bef9f

This comment was marked as outdated.

Sign in to view

edgao added 2 commits May 4, 2023 16:35

whoops

865b153

bump metadata

731d03f

bump metadatas

d085292

octavia-squidington-iii added 5 commits May 5, 2023 15:39

auto-bump connector version

536f61a

auto-bump connector version

198b1a1

auto-bump connector version

591eeeb

auto-bump connector version

2de772e

auto-bump connector version

6d8512a

edgao enabled auto-merge (squash) May 5, 2023 18:56

edgao merged commit ccb6a07 into master May 5, 2023

edgao deleted the edgao/dest_s3_ints branch May 5, 2023 19:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avro destinations: map airbyte_type: integer to long #25570

Avro destinations: map airbyte_type: integer to long #25570

edgao commented Apr 26, 2023 •

edited

Loading

This comment was marked as outdated.

github-actions bot commented Apr 26, 2023 •

edited

Loading

✅ Sources (0)

✅ Destinations (10)

👀 Other Modules (1)

Actionable Items

This comment was marked as outdated.

evantahler left a comment

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

edgao commented Apr 28, 2023 •

edited by github-actions bot

Loading

edgao commented Apr 28, 2023 •

edited by github-actions bot

Loading

edgao commented Apr 28, 2023 •

edited by github-actions bot

Loading

edgao commented Apr 28, 2023 •

edited by github-actions bot

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

edgao commented May 5, 2023 •

edited by github-actions bot

Loading

edgao commented May 5, 2023 •

edited by github-actions bot

Loading

tybernstein commented May 10, 2023

edgao commented May 10, 2023

Avro destinations: map airbyte_type: integer to long #25570

Avro destinations: map airbyte_type: integer to long #25570

Conversation

edgao commented Apr 26, 2023 • edited Loading

What

This comment was marked as outdated.

github-actions bot commented Apr 26, 2023 • edited Loading

Affected Connector Report

✅ Sources (0)

✅ Destinations (10)

👀 Other Modules (1)

Actionable Items

This comment was marked as outdated.

evantahler left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

edgao commented Apr 28, 2023 • edited by github-actions bot Loading

Build Passed

edgao commented Apr 28, 2023 • edited by github-actions bot Loading

Build Passed

edgao commented Apr 28, 2023 • edited by github-actions bot Loading

Build Passed

edgao commented Apr 28, 2023 • edited by github-actions bot Loading

Build Passed

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

edgao commented May 5, 2023 • edited by github-actions bot Loading

Build Passed

edgao commented May 5, 2023 • edited by github-actions bot Loading

tybernstein commented May 10, 2023

edgao commented May 10, 2023

edgao commented Apr 26, 2023 •

edited

Loading

github-actions bot commented Apr 26, 2023 •

edited

Loading

edgao commented Apr 28, 2023 •

edited by github-actions bot

Loading

edgao commented Apr 28, 2023 •

edited by github-actions bot

Loading

edgao commented Apr 28, 2023 •

edited by github-actions bot

Loading

edgao commented Apr 28, 2023 •

edited by github-actions bot

Loading

edgao commented May 5, 2023 •

edited by github-actions bot

Loading

edgao commented May 5, 2023 •

edited by github-actions bot

Loading