🎉 Destination s3 / gcs: add option for uncompressed csv and jsonl format #12167

tuliren · 2022-04-20T09:31:18Z

What

Resolve Add option for S3 and GCS connectors to not compress CSV and JSONL formats #12001.
In recent S3 and GCS destination connectors, the serialized buffering strategy was introduced to improve the scalability of these connectors. However, the new versions also introduced breaking changes that automatically compress CSV and JSONL formats.
The change does not affect staging destinations, as staging CSV and JSONL files will remain compressed for efficiency.

How

A compression field is added to let the users choose whether they want the CSV or JSONL files to be compressed.

🚨 User Impact 🚨

S3 and GCS users now can choose to not compress the CSV and JSONL formats.

tuliren · 2022-04-20T09:31:51Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2194831891
❌ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2194831891
🐛

tuliren · 2022-04-20T09:31:56Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2194832336
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2194832336
No Python unittests run

tuliren · 2022-04-20T09:49:46Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2194922786
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2194922786
No Python unittests run

edgao

LGTM pending two comments

airbyte-integrations/connectors/destination-gcs/src/main/resources/spec.json

...ntegration/java/io/airbyte/integrations/destination/gcs/GcsCsvDestinationAcceptanceTest.java

...-integration/java/io/airbyte/integrations/destination/s3/S3CsvDestinationAcceptanceTest.java

tuliren · 2022-04-22T05:43:46Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2205946324
❌ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2205946324
🐛

tuliren · 2022-04-22T05:43:51Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2205946622
❌ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2205946622
🐛

tuliren · 2022-04-22T05:44:09Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2205947555
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2205947555
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     31    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           34      7    79%
normalization/transform_catalog/dbt_macro.py                                                                                       22      7    68%
normalization/transform_catalog/catalog_processor.py                                                                              147     80    46%
normalization/transform_catalog/transform.py                                                                                       61     38    38%
normalization/transform_catalog/stream_processor.py                                                                               534    345    35%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1447    550    62%

tuliren · 2022-04-22T05:44:17Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2205948078
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2205948078
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     31    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           34      7    79%
normalization/transform_catalog/dbt_macro.py                                                                                       22      7    68%
normalization/transform_catalog/catalog_processor.py                                                                              147     80    46%
normalization/transform_catalog/transform.py                                                                                       61     38    38%
normalization/transform_catalog/stream_processor.py                                                                               534    345    35%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1447    550    62%

tuliren · 2022-04-22T06:42:01Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2206160148
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2206160148
No Python unittests run

tuliren · 2022-04-22T06:42:06Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2206160312
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2206160312
No Python unittests run

tuliren · 2022-04-22T20:25:35Z

/publish connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2209883900
❌ Failed to publish connectors/destination-s3
❌ Couldn't auto-bump version for connectors/destination-s3

tuliren · 2022-04-22T22:15:10Z

/publish connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2210297895
🚀 Successfully published connectors/destination-s3
🚀 Auto-bumped version for connectors/destination-s3
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2210297895

tuliren · 2022-04-22T22:15:14Z

/publish connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2210298193
🚀 Successfully published connectors/destination-gcs
❌ Couldn't auto-bump version for connectors/destination-gcs

…mat (#12167) * Add gzip compression option * Add file extension method to s3 format config * Pass gzip compression to serialized buffer * Add unit test * Format code * Update integration test * Bump version and update doc * Fix unit test * Add extra gzip tests for csv and jsonl * Make compression an oneOf param * Migrate csv config to new compression spec * Migrate jsonl config to new compression spec * Update docs * Fix unit test * Fix integration tests * Format code * Bump version * auto-bump connector version * Bump gcs version in seed Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

tuliren added 6 commits April 20, 2022 01:56

Add gzip compression option

d775644

Add file extension method to s3 format config

49bc7d4

Pass gzip compression to serialized buffer

228a236

Add unit test

b2ed143

Format code

5dc6a37

Update integration test

7b36f3b

github-actions bot added the area/connectors Connector related issues label Apr 20, 2022

tuliren added 2 commits April 20, 2022 02:42

Bump version and update doc

1c2327a

Fix unit test

f5f911a

github-actions bot added the area/documentation Improvements or additions to documentation label Apr 20, 2022

tuliren requested review from edgao and noahkawasaki-airbyte April 20, 2022 14:53

edgao approved these changes Apr 20, 2022

View reviewed changes

airbyte-integrations/connectors/destination-gcs/src/main/resources/spec.json Outdated Show resolved Hide resolved

...ntegration/java/io/airbyte/integrations/destination/gcs/GcsCsvDestinationAcceptanceTest.java Outdated Show resolved Hide resolved

noahkawasaki-airbyte reviewed Apr 20, 2022

View reviewed changes

...-integration/java/io/airbyte/integrations/destination/s3/S3CsvDestinationAcceptanceTest.java Outdated Show resolved Hide resolved

tuliren added 6 commits April 20, 2022 15:52

Merge branch 'master' into liren/add-no-compression-option

cf8b6aa

Add extra gzip tests for csv and jsonl

4fb5eb5

Make compression an oneOf param

a0055d8

Migrate csv config to new compression spec

74e3b6d

Migrate jsonl config to new compression spec

8c221d6

Update docs

e00b81f

tuliren added 2 commits April 21, 2022 23:38

Fix unit test

3e26f3a

Fix integration tests

b91f468

Format code

0a383ce

tuliren added 2 commits April 22, 2022 15:13

Merge branch 'master' into liren/add-no-compression-option

a96d425

Bump version

8502b24

auto-bump connector version

ea4664e

octavia-squidington-iii temporarily deployed to more-secrets April 22, 2022 22:33 Inactive

Bump gcs version in seed

d68882f

tuliren merged commit 9a0442c into master Apr 22, 2022

tuliren deleted the liren/add-no-compression-option branch April 22, 2022 22:38

tuliren temporarily deployed to more-secrets April 22, 2022 22:39 Inactive

octavia-squidington-iii mentioned this pull request Apr 25, 2022

Bump Airbyte version from 0.36.2-alpha to 0.36.3-alpha #12306

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Destination s3 / gcs: add option for uncompressed csv and jsonl format #12167

🎉 Destination s3 / gcs: add option for uncompressed csv and jsonl format #12167

tuliren commented Apr 20, 2022 •

edited

Loading

tuliren commented Apr 20, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 20, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 20, 2022 •

edited by github-actions bot

Loading

edgao left a comment

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

🎉 Destination s3 / gcs: add option for uncompressed csv and jsonl format #12167

🎉 Destination s3 / gcs: add option for uncompressed csv and jsonl format #12167

Conversation

tuliren commented Apr 20, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

tuliren commented Apr 20, 2022 • edited by github-actions bot Loading

tuliren commented Apr 20, 2022 • edited by github-actions bot Loading

tuliren commented Apr 20, 2022 • edited by github-actions bot Loading

edgao left a comment

Choose a reason for hiding this comment

tuliren commented Apr 22, 2022 • edited by github-actions bot Loading

tuliren commented Apr 22, 2022 • edited by github-actions bot Loading

tuliren commented Apr 22, 2022 • edited by github-actions bot Loading

tuliren commented Apr 22, 2022 • edited by github-actions bot Loading

tuliren commented Apr 22, 2022 • edited by github-actions bot Loading

tuliren commented Apr 22, 2022 • edited by github-actions bot Loading

tuliren commented Apr 22, 2022 • edited by github-actions bot Loading

tuliren commented Apr 22, 2022 • edited by github-actions bot Loading

tuliren commented Apr 22, 2022 • edited by github-actions bot Loading

tuliren commented Apr 20, 2022 •

edited

Loading

tuliren commented Apr 20, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 20, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 20, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 22, 2022 •

edited by github-actions bot

Loading