Destination redshift: async standard inserts #32888

edgao · 2023-11-28T18:28:21Z

Add an async version of the jdbc consumer
- it's a pretty thin wrapper around existing code - almost identical to the non-async consumer
- Delete the non-async consumer
push PartialAirbyteMessage everywhere instead of AirbyteRecordMessage
- This will be a breaking change in the cdk, but hopefully pretty easy to implement in each destination connector - e.g. redshift gets it almost for free, just a method signature update

I ran a 20M record sync (3 streams of 10M + 10M + 100 records) here https://cloud.airbyte.com/workspaces/b61bc266-ef3c-460c-af2f-f70da4a2993c/connections/3a93c4ec-9b61-42f8-b6f2-4ec052129dfc/job-history#6303716::0; it ran successfully and in approximately the same time as previous syncs. (2hr40min vs older syncs ranging from 2:28 - 3:43)

vercel · 2023-11-28T18:28:27Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Nov 30, 2023 4:28pm

github-actions · 2023-11-28T18:28:41Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

edgao · 2023-11-29T17:22:54Z

...c/main/java/io/airbyte/cdk/integrations/base/spec_modification/SpecModifyingDestination.java

@@ -41,4 +42,12 @@ public AirbyteMessageConsumer getConsumer(final JsonNode config,
    return destination.getConsumer(config, catalog, outputRecordCollector);
  }

+  @Override
+  public SerializedAirbyteMessageConsumer getSerializedMessageConsumer(final JsonNode config,


this isn't used by redshift, just adding it since I'm in this part of the code anyway

edgao · 2023-11-29T17:41:59Z

.../src/main/java/io/airbyte/cdk/integrations/destination/jdbc/JdbcBufferedConsumerFactory.java

        catalog,
-        sqlOperations::isValidData);


every implementation of this method is just return true 🤷 but deleting it looks like a nontrivial refactor so ignoring it for now. It's not used in the async framework at all, and we don't want this functionality to begin with - it's dropping "invalid" records, which we don't want to do.

edgao · 2023-11-29T17:43:11Z

...stinations/src/main/java/io/airbyte/cdk/integrations/destination/jdbc/JdbcSqlOperations.java

                                  final String schemaName,
                                  final String tableName)
      throws Exception {
-    dataAdapter.ifPresent(adapter -> records.forEach(airbyteRecordMessage -> adapter.adapt(airbyteRecordMessage.getData())));
+    dataAdapter.ifPresent(adapter -> records.forEach(airbyteRecordMessage -> {


the only DataAdapter in existence is destination-postgres, because of https://airbytehq-team.slack.com/archives/C03C4AVJWG4/p1700605909300539 / #3476

so I don't feel bad about de+reserializing here

edgao · 2023-11-29T17:43:49Z

...tinations/src/main/java/io/airbyte/cdk/integrations/destination/jdbc/SqlOperationsUtils.java

            // 1-indexed
            statement.setString(i, uuidSupplier.get().toString());
-            statement.setString(i + 1, Jsons.serialize(message.getData()));
-            statement.setTimestamp(i + 2, Timestamp.from(Instant.ofEpochMilli(message.getEmittedAt())));
+            statement.setString(i + 1, message.getSerialized());


note that we can skip the Jsons.serialize call here

gisripa

lgtm. pending CI tests and connector checklist.

Can you also paste the link of the workspace where you ran large sync for posterity.

edgao · 2023-11-29T23:06:22Z

added a link in the pr description. dest-redshift tests are actually succeeding, we're failing because of source-google-search-console autoformat >.>

edgao · 2023-11-29T23:10:28Z

meh. I'll publish this tomorrow. Either someone else will merge master + fix the format, or I'll just approve-and-merge

(just don't want to release this when I'm about to go on my commute)

edgao · 2023-11-30T16:08:23Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/7049134047
✅ Successfully published Java CDK version=0.6.0!

redshift async standard inserts

2a59e85

edgao requested a review from a team as a code owner November 28, 2023 18:28

octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/destination/redshift labels Nov 28, 2023

edgao and others added 7 commits November 28, 2023 18:33

Automated Commit - Formatting Changes

9fe0da3

redshift version bump

00dc110

fix batch size

57a9fd4

try lowering memory limit

b87b935

even lower

b7640d6

Merge branch 'master' into edgao/redshift/standard_inserts_async

951e8eb

changelog

3021982

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Nov 29, 2023

vercel bot deployed to Preview November 29, 2023 17:06 View deployment

edgao and others added 6 commits November 29, 2023 09:08

undo

89e2b47

also update specmodifyingdestination

617a75a

Automated Commit - Formatting Changes

1759186

stop supporting non-async consumers

cf3171f

use async + partial message everywhere

a6e37df

Automated Commit - Formatting Changes

b73a47f

edgao commented Nov 29, 2023

View reviewed changes

edgao and others added 4 commits November 29, 2023 10:09

2mb batch

949be58

Automated Commit - Formatting Changes

6b74f57

actually lets use 5mb

0067616

Merge branch 'master' into edgao/redshift/standard_inserts_async

7128786

vercel bot deployed to Preview November 29, 2023 19:32 View deployment

edgao and others added 2 commits November 29, 2023 11:36

delete unused code

5f70fd0

Automated Commit - Formatting Changes

422af2e

bnchrch and others added 3 commits November 29, 2023 13:33

Fix: Revert airbyte-ci to 2.8.0 from 2.10.0 (#32954)

90fb108

revert to 25MB batch

b42a93e

Merge branch 'master' into edgao/redshift/standard_inserts_async

3d1ff9d

vercel bot deployed to Preview November 29, 2023 21:36 View deployment

Automated Commit - Formatting Changes

2051da9

octavia-squidington-iii added the connectors/source/google-search-console label Nov 29, 2023

octavia-squidington-iv requested a review from a team November 29, 2023 21:47

gisripa approved these changes Nov 29, 2023

View reviewed changes

Merge branch 'master' into edgao/redshift/standard_inserts_async

1d2bca9

vercel bot deployed to Preview November 30, 2023 16:07 View deployment

cdk version bump

f4daa6a

edgao added 2 commits November 30, 2023 08:10

switch back to published cdk artifact

b9b246f

Merge branch 'master' into edgao/redshift/standard_inserts_async

95c21d0

edgao enabled auto-merge (squash) November 30, 2023 16:35

edgao merged commit 6864074 into master Nov 30, 2023
21 checks passed

edgao deleted the edgao/redshift/standard_inserts_async branch November 30, 2023 16:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Destination redshift: async standard inserts #32888

Destination redshift: async standard inserts #32888

edgao commented Nov 28, 2023 •

edited

Loading

vercel bot commented Nov 28, 2023 •

edited

Loading

github-actions bot commented Nov 28, 2023 •

edited by edgao

Loading

edgao Nov 29, 2023

edgao Nov 29, 2023

edgao Nov 29, 2023

edgao Nov 29, 2023

gisripa left a comment

edgao commented Nov 29, 2023

edgao commented Nov 29, 2023

edgao commented Nov 30, 2023 •

edited by github-actions bot

Loading

Destination redshift: async standard inserts #32888

Destination redshift: async standard inserts #32888

Conversation

edgao commented Nov 28, 2023 • edited Loading

vercel bot commented Nov 28, 2023 • edited Loading

github-actions bot commented Nov 28, 2023 • edited by edgao Loading

Before Merging a Connector Pull Request

edgao Nov 29, 2023

Choose a reason for hiding this comment

edgao Nov 29, 2023

Choose a reason for hiding this comment

edgao Nov 29, 2023

Choose a reason for hiding this comment

edgao Nov 29, 2023

Choose a reason for hiding this comment

gisripa left a comment

Choose a reason for hiding this comment

edgao commented Nov 29, 2023

edgao commented Nov 29, 2023

edgao commented Nov 30, 2023 • edited by github-actions bot Loading

edgao commented Nov 28, 2023 •

edited

Loading

vercel bot commented Nov 28, 2023 •

edited

Loading

github-actions bot commented Nov 28, 2023 •

edited by edgao

Loading

edgao commented Nov 30, 2023 •

edited by github-actions bot

Loading