✨Switch redshift staging to async mode #28619

benmoriceau · 2023-07-24T16:15:07Z

What

Switch the redshift staging to the async framework.

How

Use the createAsync method instead of the create to get an AirbyteMessage consumer. This Pr also provide the ability to specify a different Optimal flush size for the Async flush. Even if it is not needed to release readshift, it was used for some test and kept into this PR.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user?

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

github-actions · 2023-07-24T16:15:35Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

benmoriceau · 2023-07-24T16:35:18Z

/legacy-test connector=destination-redshift

🕑 destination-redshift https://github.com/airbytehq/airbyte/actions/runs/5647455583
❌ destination-redshift https://github.com/airbytehq/airbyte/actions/runs/5647455583
🐛 https://gradle.com/s/4vsvz6oqczdm4

Build Failed

Test summary info:

Could not find result summary

…nc-snowflake

… into bmoric/async-redshift

benmoriceau · 2023-07-26T17:58:43Z

/legacy-test connector=destination-redshift

🕑 destination-redshift https://github.com/airbytehq/airbyte/actions/runs/5672127756
❌ destination-redshift https://github.com/airbytehq/airbyte/actions/runs/5672127756
🐛

…into bmoric/async-redshift

…nc-redshift

octavia-squidington-iii · 2023-08-10T20:24:09Z

destination-redshift test report (commit `d3357da880`) - ❌

⏲️ Total pipeline duration: 03mn03s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

octavia-squidington-iii · 2023-08-10T20:50:06Z

destination-redshift test report (commit `78a70dbd58`) - ❌

⏲️ Total pipeline duration: 24mn23s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

…nc-redshift

octavia-squidington-iii · 2023-08-11T15:57:18Z

destination-redshift test report (commit `7de0a1bebc`) - ❌

⏲️ Total pipeline duration: 25mn50s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

…nc-redshift

octavia-squidington-iii · 2023-08-14T15:09:44Z

destination-redshift test report (commit `e2b54dda80`) - ❌

⏲️ Total pipeline duration: 22mn29s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

octavia-squidington-iii · 2023-08-14T19:15:47Z

destination-redshift test report (commit `0fa168cd47`) - ❌

⏲️ Total pipeline duration: 81mn12s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

...redshift/src/main/java/io/airbyte/integrations/destination/redshift/RedshiftDestination.java

davinchia · 2023-08-14T20:12:24Z

...src/main/java/io/airbyte/integrations/destination/redshift/RedshiftStagingS3Destination.java

    final JsonNode s3Options = findS3Options(config);
    final S3DestinationConfig s3Config = getS3DestinationConfig(s3Options);
    final int numberOfFileBuffers = getNumberOfFileBuffers(s3Options);
-
    if (numberOfFileBuffers > FileBuffer.SOFT_CAP_CONCURRENT_STREAM_IN_BUFFER) {


We can also get rid of the file buffers option since the Async framework accounts for this. I think we should do so in a follow up PR and patch version bump. @edgao any preference?

If Edward is ok, I can do it after merging this

followup pr works for me!

...s-destination-jdbc/src/main/java/io/airbyte/integrations/destination/staging/AsyncFlush.java

…a/io/airbyte/integrations/destination/staging/AsyncFlush.java Co-authored-by: Davin Chia <davinchia@gmail.com>

edgao

approving with failing checks because tests are probably just busted on CI. E.g. testIncrementalSyncWithNormalizationDropOneColumn failed here, but passed locally for benoit.

benmoriceau · 2023-08-14T20:52:29Z

/approve-and-merge reason="local test works"

octavia-approvington · 2023-08-14T20:53:03Z

What are we doing again?

octavia-squidington-iii · 2023-08-14T21:03:47Z

destination-redshift test report (commit `2da179d22c`) - ❌

⏲️ Total pipeline duration: 53mn27s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

octavia-squidington-iii · 2023-08-14T22:01:49Z

destination-redshift test report (commit `3dd8a40b3d`) - ❌

⏲️ Total pipeline duration: 55mn46s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

* Async snowflake * Use async in destination implenentation * Format * Switch redshif to asyn mode * Remove old unused consumer creation * Add new version * Fix non staging mode * Change switcing to use the get serialized consumer * Automated Commit - Format and Process Resources Changes * Test * Automated Commit - Format and Process Resources Changes * Use method * Test smaller buffer * Test smaller buffer for redshift * Automated Commit - Format and Process Resources Changes * Bigger ratio * Remove snowflake changes * Implement the new interface * Automated Commit - Format and Process Resources Changes * push ratio to 0.8 * Smaller Optimal buffer size * Automated Commit - Format and Process Resources Changes * Bigger buffer * Use a buffer of 10 Mb * Use a buffer of 75 Mb * Test reduce lib thread * Add flags for remote profiler. * Part size to match the async part size * Part size to 100 Mb * restore default * Try with 1 thread * Go back to default * Clean up * Bump version * Restore gradle * Re-add vm capture * Test reduce allowed buffer size * Use all the memory available * only 3 threads for the lib * Automated Commit - Format and Process Resources Changes * test with 1 * Automated Commit - Format and Process Resources Changes * Add local log ling. * Do not use all RAM for heap. * Fix build * Clean up * Clean up * Update airbyte-integrations/bases/bases-destination-jdbc/src/main/java/io/airbyte/integrations/destination/staging/AsyncFlush.java Co-authored-by: Davin Chia <davinchia@gmail.com> * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>

benmoriceau and others added 6 commits July 19, 2023 15:47

Async snowflake

e6f0ba0

Use async in destination implenentation

0ce1aac

Format

51fbe49

Merge branch 'master' into bmoric/async-snowflake

4263a79

Merge branch 'master' into bmoric/async-snowflake

978cfd2

Switch redshif to asyn mode

5aa307b

octavia-squidington-iii added area/connectors Connector related issues connectors/destination/redshift labels Jul 24, 2023

benmoriceau added 4 commits July 24, 2023 14:02

Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/asy…

6a083d1

…nc-snowflake

Remove old unused consumer creation

b1b9343

Add new version

fd8acfb

Merge branch 'bmoric/async-snowflake' of github.com:airbytehq/airbyte…

57844fd

… into bmoric/async-redshift

benmoriceau requested review from davinchia and tryangul July 25, 2023 17:28

benmoriceau added 4 commits July 25, 2023 12:52

Merge branch 'master' into bmoric/async-snowflake

5123f87

Merge branch 'master' into bmoric/async-snowflake

f15ca3f

Merge branch 'master' into bmoric/async-snowflake

e5745b9

Merge branch 'bmoric/async-snowflake' into bmoric/async-redshift

e8bb0f4

benmoriceau added 2 commits July 27, 2023 08:31

Fix non staging mode

c01e7ef

Change switcing to use the get serialized consumer

7a1ae97

benmoriceau changed the title ~~Switch redshif to asyn mode~~ Switch redshift to asyn mode Jul 27, 2023

benmoriceau and others added 6 commits July 27, 2023 19:08

Automated Commit - Format and Process Resources Changes

73643b3

Test

f110532

Merge branch 'bmoric/async-redshift' of github.com:airbytehq/airbyte …

ccada79

…into bmoric/async-redshift

Automated Commit - Format and Process Resources Changes

cd30db3

Use method

a707c68

Merge branch 'bmoric/async-redshift' of github.com:airbytehq/airbyte …

36df2a1

…into bmoric/async-redshift

benmoriceau added 2 commits August 10, 2023 13:19

Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/asy…

d3357da

…nc-redshift

Fix build

78a70db

Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/asy…

7de0a1b

…nc-redshift

benmoriceau added 2 commits August 11, 2023 11:32

Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/asy…

e2b54dd

…nc-redshift

Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/asy…

c96d32c

…nc-redshift

Clean up

0fa168c

edgao reviewed Aug 14, 2023

View reviewed changes

...redshift/src/main/java/io/airbyte/integrations/destination/redshift/RedshiftDestination.java Outdated Show resolved Hide resolved

Clean up

2da179d

davinchia reviewed Aug 14, 2023

View reviewed changes

...s-destination-jdbc/src/main/java/io/airbyte/integrations/destination/staging/AsyncFlush.java Outdated Show resolved Hide resolved

Update airbyte-integrations/bases/bases-destination-jdbc/src/main/jav…

f2feec4

…a/io/airbyte/integrations/destination/staging/AsyncFlush.java Co-authored-by: Davin Chia <davinchia@gmail.com>

benmoriceau requested review from edgao and davinchia August 14, 2023 20:25

edgao approved these changes Aug 14, 2023

View reviewed changes

Automated Commit - Format and Process Resources Changes

97ac3fb

octavia-approvington approved these changes Aug 14, 2023

View reviewed changes

octavia-approvington merged commit 8d19017 into master Aug 14, 2023

octavia-approvington deleted the bmoric/async-redshift branch August 14, 2023 20:53

sh4sh mentioned this pull request Oct 3, 2023

Destination Redshift: Connector v0.6.5 runs out of memory #30995

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨Switch redshift staging to async mode #28619

✨Switch redshift staging to async mode #28619

benmoriceau commented Jul 24, 2023 •

edited

Loading

github-actions bot commented Jul 24, 2023

benmoriceau commented Jul 24, 2023 •

edited by github-actions bot

Loading

benmoriceau commented Jul 26, 2023 •

edited by github-actions bot

Loading

octavia-squidington-iii commented Aug 10, 2023

octavia-squidington-iii commented Aug 10, 2023

octavia-squidington-iii commented Aug 11, 2023

octavia-squidington-iii commented Aug 14, 2023

octavia-squidington-iii commented Aug 14, 2023

davinchia Aug 14, 2023

benmoriceau Aug 14, 2023

edgao Aug 14, 2023 •

edited

Loading

edgao left a comment

benmoriceau commented Aug 14, 2023

octavia-approvington commented Aug 14, 2023

octavia-squidington-iii commented Aug 14, 2023

octavia-squidington-iii commented Aug 14, 2023

✨Switch redshift staging to async mode #28619

✨Switch redshift staging to async mode #28619

Conversation

benmoriceau commented Jul 24, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

github-actions bot commented Jul 24, 2023

Before Merging a Connector Pull Request

benmoriceau commented Jul 24, 2023 • edited by github-actions bot Loading

Build Failed

benmoriceau commented Jul 26, 2023 • edited by github-actions bot Loading

octavia-squidington-iii commented Aug 10, 2023

destination-redshift test report (commit d3357da880) - ❌

octavia-squidington-iii commented Aug 10, 2023

destination-redshift test report (commit 78a70dbd58) - ❌

octavia-squidington-iii commented Aug 11, 2023

destination-redshift test report (commit 7de0a1bebc) - ❌

octavia-squidington-iii commented Aug 14, 2023

destination-redshift test report (commit e2b54dda80) - ❌

octavia-squidington-iii commented Aug 14, 2023

destination-redshift test report (commit 0fa168cd47) - ❌

davinchia Aug 14, 2023

Choose a reason for hiding this comment

benmoriceau Aug 14, 2023

Choose a reason for hiding this comment

edgao Aug 14, 2023 • edited Loading

Choose a reason for hiding this comment

edgao left a comment

Choose a reason for hiding this comment

benmoriceau commented Aug 14, 2023

octavia-approvington commented Aug 14, 2023

octavia-squidington-iii commented Aug 14, 2023

destination-redshift test report (commit 2da179d22c) - ❌

octavia-squidington-iii commented Aug 14, 2023

destination-redshift test report (commit 3dd8a40b3d) - ❌

benmoriceau commented Jul 24, 2023 •

edited

Loading

benmoriceau commented Jul 24, 2023 •

edited by github-actions bot

Loading

benmoriceau commented Jul 26, 2023 •

edited by github-actions bot

Loading

destination-redshift test report (commit `d3357da880`) - ❌

destination-redshift test report (commit `78a70dbd58`) - ❌

destination-redshift test report (commit `7de0a1bebc`) - ❌

destination-redshift test report (commit `e2b54dda80`) - ❌

destination-redshift test report (commit `0fa168cd47`) - ❌

edgao Aug 14, 2023 •

edited

Loading

destination-redshift test report (commit `2da179d22c`) - ❌

destination-redshift test report (commit `3dd8a40b3d`) - ❌