Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Relax Type and Dedupe frequency #28758

Merged
merged 11 commits into from
Aug 1, 2023
Merged

Relax Type and Dedupe frequency #28758

merged 11 commits into from
Aug 1, 2023

Conversation

evantahler
Copy link
Contributor

This PR relaxes the frequency that we run type and dedupe for Destinations V2.

New users of airbyte likely want to see data flowing into their tables as soon as possible, and we want to catch new errors which might appear early within an incremental sync.
However, as their destination tables grow in size, typing and de-duping data becomes an expensive operation.
To strike a balance between showing data quickly and not slowing down the entire sync, we use an increasing interval based approach, from 0 up to 4 hours.
This is not fancy, just hard coded intervals.

evantahler
evantahler commented Jul 26, 2023
View reviewed changes
...o/airbyte/integrations/base/destination/typing_deduping/TypeAndDedupeOperationValveTest.java
Comment on lines 66 to 77
@Test
public void testIncrementInterval() {
final var valve = new TypeAndDedupeOperationValve(ALWAYS_ZERO);
valve.addStream(STREAM_A);
IntStream.rangeClosed(1, 3).forEach(i -> {
IntStream.rangeClosed(1, 4).forEach(i -> {
final var index = valve.incrementInterval(STREAM_A);
Assertions.assertEquals(i, index);
});
Assertions.assertEquals(3, valve.incrementInterval(STREAM_A));
Assertions.assertEquals(4, valve.incrementInterval(STREAM_A));
// Twice to be sure
Assertions.assertEquals(3, valve.incrementInterval(STREAM_A));
Assertions.assertEquals(4, valve.incrementInterval(STREAM_A));
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jbfbell I'll be honest... I don't really understand what this test is doing. But, I added one more increment of time, so the numbers all +1 seems right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah probably should add a comment explaining - its testing that as you increment the interval, its at that expected interval, (if you're at 1 minute, when you increment you're now at 15 minutes) but with a cap so if you increment when you're already at the last index, you stay there.

But yes your change is correct

@evantahler evantahler marked this pull request as ready for review July 26, 2023 22:38
@evantahler evantahler requested a review from jbfbell July 26, 2023 22:38
@octavia-squidington-iii
Copy link
Collaborator

destination-tidb test report (commit 325c485981) - ✅

⏲️ Total pipeline duration: 26mn57s

Step Result
Validate airbyte-integrations/connectors/destination-tidb/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-tidb docker image for platform linux/x86_64 ✅
Build airbyte/normalization-tidb:dev ✅
./gradlew :airbyte-integrations:connectors:destination-tidb:integrationTest ✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-tidb test

@octavia-squidington-iii
Copy link
Collaborator

destination-mysql test report (commit 325c485981) - ❌

⏲️ Total pipeline duration: 54mn20s

Step Result
Validate airbyte-integrations/connectors/destination-mysql/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mysql docker image for platform linux/x86_64 ✅
Build airbyte/normalization-mysql:dev ✅
./gradlew :airbyte-integrations:connectors:destination-mysql:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql test

@octavia-squidington-iii
Copy link
Collaborator

destination-databricks test report (commit 325c485981) - ❌

⏲️ Total pipeline duration: 53mn36s

Step Result
Validate airbyte-integrations/connectors/destination-databricks/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-databricks docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-databricks:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-databricks test

@evantahler evantahler requested a review from a team July 27, 2023 18:10
jbfbell
jbfbell approved these changes Jul 27, 2023
View reviewed changes
Copy link
Contributor

@jbfbell jbfbell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I think there is probably an associated change required to achieve what we discussed in the normalization group meeting yesterday. I think the changes would be:

  • Change ONE_MINUTE_MILLIS to zero or negative (and a matching name)
  • Change this conditional in BigQueryStagingConsumerFactory from an else if to an if so we add the stream and then it is immediately ready to type and dedupe

We can tackle that in a separate PR if necessary

...o/airbyte/integrations/base/destination/typing_deduping/TypeAndDedupeOperationValveTest.java
Comment on lines 66 to 77
@Test
public void testIncrementInterval() {
final var valve = new TypeAndDedupeOperationValve(ALWAYS_ZERO);
valve.addStream(STREAM_A);
IntStream.rangeClosed(1, 3).forEach(i -> {
IntStream.rangeClosed(1, 4).forEach(i -> {
final var index = valve.incrementInterval(STREAM_A);
Assertions.assertEquals(i, index);
});
Assertions.assertEquals(3, valve.incrementInterval(STREAM_A));
Assertions.assertEquals(4, valve.incrementInterval(STREAM_A));
// Twice to be sure
Assertions.assertEquals(3, valve.incrementInterval(STREAM_A));
Assertions.assertEquals(4, valve.incrementInterval(STREAM_A));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah probably should add a comment explaining - its testing that as you increment the interval, its at that expected interval, (if you're at 1 minute, when you increment you're now at 15 minutes) but with a cap so if you increment when you're already at the last index, you stay there.

But yes your change is correct

@octavia-squidington-iii
Copy link
Collaborator

destination-teradata test report (commit 12688aac49) - ✅

⏲️ Total pipeline duration: 17mn53s

Step Result
Validate airbyte-integrations/connectors/destination-teradata/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-teradata docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-teradata:integrationTest ✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-teradata test

@octavia-squidington-iii octavia-squidington-iii added the area/connectors Connector related issues label Jul 28, 2023
@evantahler
Copy link
Contributor Author

@jbfbell I made those changes in d9d0c03

@evantahler evantahler requested a review from jbfbell July 28, 2023 00:00
evantahler
evantahler commented Jul 28, 2023
View reviewed changes
...c/main/java/io/airbyte/integrations/destination/bigquery/BigQueryStagingConsumerFactory.java Outdated
} else if (valve.readyToTypeAndDedupe(streamId)) {
} if (valve.readyToTypeAndDedupe(streamId)) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to @edgao - maybe a similar thing needs to change for Snowflake

Copy link
Contributor

@edgao edgao Jul 31, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah is this because we have to type+dedupe the first time we see each stream?

also style nit, put the if on a new line: (i'm surprised the linter didn't catch this)

} 
if (valve.readyToTypeAndDedupe(streamId)) {

(fixed for snowflake in 68ae5f0)

@octavia-squidington-iii
Copy link
Collaborator

destination-clickhouse-strict-encrypt test report (commit 12688aac49) - ❌

⏲️ Total pipeline duration: 305mn41s

Step Result
Validate airbyte-integrations/connectors/destination-clickhouse-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-clickhouse-strict-encrypt docker image for platform linux/x86_64 ✅
Build airbyte/normalization-clickhouse:dev ✅
./gradlew :airbyte-integrations:connectors:destination-clickhouse-strict-encrypt:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-clickhouse-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-mysql-strict-encrypt test report (commit 12688aac49) - ❌

⏲️ Total pipeline duration: 38mn33s

Step Result
Validate airbyte-integrations/connectors/destination-mysql-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mysql-strict-encrypt docker image for platform linux/x86_64 ✅
Build airbyte/normalization-mysql:dev ✅
./gradlew :airbyte-integrations:connectors:destination-mysql-strict-encrypt:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-oracle-strict-encrypt test report (commit 12688aac49) - ❌

⏲️ Total pipeline duration: 34mn33s

Step Result
Validate airbyte-integrations/connectors/destination-oracle-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-oracle-strict-encrypt docker image for platform linux/x86_64 ✅
Build airbyte/normalization-oracle:dev ✅
./gradlew :airbyte-integrations:connectors:destination-oracle-strict-encrypt:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-oracle-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-snowflake test report (commit 12688aac49) - ✅

⏲️ Total pipeline duration: 61mn32s

Step Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-snowflake docker image for platform linux/x86_64 ✅
Build airbyte/normalization-snowflake:dev ✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest ✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@octavia-squidington-iii
Copy link
Collaborator

destination-postgres-strict-encrypt test report (commit 12688aac49) - ❌

⏲️ Total pipeline duration: 20mn38s

Step Result
Validate airbyte-integrations/connectors/destination-postgres-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-postgres-strict-encrypt docker image for platform linux/x86_64 ✅
Build airbyte/normalization:dev ✅
./gradlew :airbyte-integrations:connectors:destination-postgres-strict-encrypt:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-postgres-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-mariadb-columnstore test report (commit 12688aac49) - ❌

⏲️ Total pipeline duration: 96mn02s

Step Result
Validate airbyte-integrations/connectors/destination-mariadb-columnstore/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mariadb-columnstore docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-mariadb-columnstore:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mariadb-columnstore test

@octavia-squidington-iii
Copy link
Collaborator

destination-mssql-strict-encrypt test report (commit 12688aac49) - ❌

⏲️ Total pipeline duration: 31mn57s

Step Result
Validate airbyte-integrations/connectors/destination-mssql-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mssql-strict-encrypt docker image for platform linux/x86_64 ✅
Build airbyte/normalization-mssql:dev ✅
./gradlew :airbyte-integrations:connectors:destination-mssql-strict-encrypt:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mssql-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-bigquery-denormalized test report (commit 12688aac49) - ✅

⏲️ Total pipeline duration: 24mn02s

Step Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest ✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

@octavia-squidington-iii
Copy link
Collaborator

destination-azure-blob-storage test report (commit 12688aac49) - ❌

⏲️ Total pipeline duration: 14mn02s

Step Result
Validate airbyte-integrations/connectors/destination-azure-blob-storage/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-azure-blob-storage docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-azure-blob-storage:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-azure-blob-storage test

@octavia-squidington-iii
Copy link
Collaborator

destination-clickhouse test report (commit 12688aac49) - ❌

⏲️ Total pipeline duration: 305mn55s

Step Result
Validate airbyte-integrations/connectors/destination-clickhouse/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-clickhouse docker image for platform linux/x86_64 ✅
Build airbyte/normalization-clickhouse:dev ✅
./gradlew :airbyte-integrations:connectors:destination-clickhouse:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-clickhouse test

@octavia-squidington-iii
Copy link
Collaborator

destination-clickhouse-strict-encrypt test report (commit 75824dd8c4) - ❌

⏲️ Total pipeline duration: 04mn09s

Step Result
Validate airbyte-integrations/connectors/destination-clickhouse-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-clickhouse-strict-encrypt docker image for platform linux/x86_64 ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-clickhouse-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-mysql test report (commit 75824dd8c4) - ❌

⏲️ Total pipeline duration: 04mn11s

Step Result
Validate airbyte-integrations/connectors/destination-mysql/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mysql docker image for platform linux/x86_64 ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql test

@octavia-squidington-iii
Copy link
Collaborator

destination-bigquery-denormalized test report (commit 75824dd8c4) - ❌

⏲️ Total pipeline duration: 04mn23s

Step Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64 ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

@octavia-squidington-iii
Copy link
Collaborator

destination-postgres-strict-encrypt test report (commit 75824dd8c4) - ❌

⏲️ Total pipeline duration: 04mn03s

Step Result
Validate airbyte-integrations/connectors/destination-postgres-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-postgres-strict-encrypt docker image for platform linux/x86_64 ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-postgres-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-mysql-strict-encrypt test report (commit 75824dd8c4) - ❌

⏲️ Total pipeline duration: 04mn04s

Step Result
Validate airbyte-integrations/connectors/destination-mysql-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mysql-strict-encrypt docker image for platform linux/x86_64 ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-dynamodb test report (commit 75824dd8c4) - ❌

⏲️ Total pipeline duration: 04mn15s

Step Result
Validate airbyte-integrations/connectors/destination-dynamodb/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-dynamodb docker image for platform linux/x86_64 ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-dynamodb test

@octavia-squidington-iii
Copy link
Collaborator

destination-clickhouse test report (commit 75824dd8c4) - ❌

⏲️ Total pipeline duration: 04mn11s

Step Result
Validate airbyte-integrations/connectors/destination-clickhouse/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-clickhouse docker image for platform linux/x86_64 ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-clickhouse test

@octavia-squidington-iii
Copy link
Collaborator

destination-dynamodb test report (commit 8b74aa3a7f) - ✅

⏲️ Total pipeline duration: 20mn17s

Step Result
Validate airbyte-integrations/connectors/destination-dynamodb/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-dynamodb docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-dynamodb:integrationTest ✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-dynamodb test

@octavia-squidington-iii
Copy link
Collaborator

destination-mysql-strict-encrypt test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 41mn18s

Step Result
Validate airbyte-integrations/connectors/destination-mysql-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mysql-strict-encrypt docker image for platform linux/x86_64 ✅
Build airbyte/normalization-mysql:dev ✅
./gradlew :airbyte-integrations:connectors:destination-mysql-strict-encrypt:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-yugabytedb test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 28mn44s

Step Result
Validate airbyte-integrations/connectors/destination-yugabytedb/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-yugabytedb docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-yugabytedb:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-yugabytedb test

@octavia-squidington-iii
Copy link
Collaborator

destination-clickhouse test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 305mn20s

Step Result
Validate airbyte-integrations/connectors/destination-clickhouse/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-clickhouse docker image for platform linux/x86_64 ✅
Build airbyte/normalization-clickhouse:dev ✅
./gradlew :airbyte-integrations:connectors:destination-clickhouse:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-clickhouse test

@octavia-squidington-iii
Copy link
Collaborator

destination-postgres test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 27mn58s

Step Result
Validate airbyte-integrations/connectors/destination-postgres/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-postgres docker image for platform linux/x86_64 ✅
Build airbyte/normalization:dev ✅
./gradlew :airbyte-integrations:connectors:destination-postgres:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-postgres test

@octavia-squidington-iii
Copy link
Collaborator

destination-redshift test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 22mn37s

Step Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-redshift docker image for platform linux/x86_64 ✅
Build airbyte/normalization-redshift:dev ✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

@octavia-squidington-iii
Copy link
Collaborator

destination-gcs test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 12mn14s

Step Result
Validate airbyte-integrations/connectors/destination-gcs/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-gcs docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-gcs:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-gcs test

@octavia-squidington-iii
Copy link
Collaborator

destination-mariadb-columnstore test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 79mn20s

Step Result
Validate airbyte-integrations/connectors/destination-mariadb-columnstore/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mariadb-columnstore docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-mariadb-columnstore:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mariadb-columnstore test

@octavia-squidington-iii
Copy link
Collaborator

destination-postgres-strict-encrypt test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 14mn12s

Step Result
Validate airbyte-integrations/connectors/destination-postgres-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-postgres-strict-encrypt docker image for platform linux/x86_64 ✅
Build airbyte/normalization:dev ✅
./gradlew :airbyte-integrations:connectors:destination-postgres-strict-encrypt:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-postgres-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-oracle test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 305mn51s

Step Result
Validate airbyte-integrations/connectors/destination-oracle/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-oracle docker image for platform linux/x86_64 ✅
Build airbyte/normalization-oracle:dev ✅
./gradlew :airbyte-integrations:connectors:destination-oracle:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-oracle test

@octavia-squidington-iii
Copy link
Collaborator

destination-teradata test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 17mn55s

Step Result
Validate airbyte-integrations/connectors/destination-teradata/metadata.yaml ✅
Connector version semver check ✅
QA checks ❌
Build connector tar ✅
Build destination-teradata docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-teradata:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-teradata test

@octavia-squidington-iii
Copy link
Collaborator

destination-rockset test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 11mn09s

Step Result
Validate airbyte-integrations/connectors/destination-rockset/metadata.yaml ✅
Connector version semver check ✅
QA checks ❌
Build connector tar ✅
Build destination-rockset docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-rockset:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-rockset test

@octavia-squidington-iii
Copy link
Collaborator

destination-mssql test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 306mn24s

Step Result
Validate airbyte-integrations/connectors/destination-mssql/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mssql docker image for platform linux/x86_64 ✅
Build airbyte/normalization-mssql:dev ✅
./gradlew :airbyte-integrations:connectors:destination-mssql:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mssql test

@octavia-squidington-iii
Copy link
Collaborator

destination-bigquery-denormalized test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 12mn46s

Step Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

@octavia-squidington-iii
Copy link
Collaborator

destination-tidb test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 17mn17s

Step Result
Validate airbyte-integrations/connectors/destination-tidb/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-tidb docker image for platform linux/x86_64 ✅
Build airbyte/normalization-tidb:dev ✅
./gradlew :airbyte-integrations:connectors:destination-tidb:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-tidb test

@octavia-squidington-iii
Copy link
Collaborator

destination-mssql-strict-encrypt test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 14mn19s

Step Result
Validate airbyte-integrations/connectors/destination-mssql-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mssql-strict-encrypt docker image for platform linux/x86_64 ✅
Build airbyte/normalization-mssql:dev ✅
./gradlew :airbyte-integrations:connectors:destination-mssql-strict-encrypt:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mssql-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-oracle-strict-encrypt test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 22mn45s

Step Result
Validate airbyte-integrations/connectors/destination-oracle-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-oracle-strict-encrypt docker image for platform linux/x86_64 ✅
Build airbyte/normalization-oracle:dev ✅
./gradlew :airbyte-integrations:connectors:destination-oracle-strict-encrypt:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-oracle-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-azure-blob-storage test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 12mn57s

Step Result
Validate airbyte-integrations/connectors/destination-azure-blob-storage/metadata.yaml ✅
Connector version semver check ✅
QA checks ❌
Build connector tar ✅
Build destination-azure-blob-storage docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-azure-blob-storage:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-azure-blob-storage test

@octavia-squidington-iii
Copy link
Collaborator

destination-mysql test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 33mn03s

Step Result
Validate airbyte-integrations/connectors/destination-mysql/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-mysql docker image for platform linux/x86_64 ✅
Build airbyte/normalization-mysql:dev ✅
./gradlew :airbyte-integrations:connectors:destination-mysql:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql test

@octavia-squidington-iii
Copy link
Collaborator

destination-bigquery test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 18mn01s

Step Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml ✅
Connector version semver check ❌
Connector version increment check ❌
QA checks ✅
Build connector tar ✅
Build destination-bigquery docker image for platform linux/x86_64 ✅
Build airbyte/normalization:dev ✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

@octavia-squidington-iii
Copy link
Collaborator

destination-exasol test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 18mn59s

Step Result
Validate airbyte-integrations/connectors/destination-exasol/metadata.yaml ✅
Connector version semver check ✅
QA checks ❌
Build connector tar ✅
Build destination-exasol docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-exasol:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-exasol test

@octavia-squidington-iii
Copy link
Collaborator

destination-clickhouse-strict-encrypt test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 06mn12s

Step Result
Validate airbyte-integrations/connectors/destination-clickhouse-strict-encrypt/metadata.yaml ✅
Connector version semver check ✅
QA checks ✅
Build connector tar ✅
Build destination-clickhouse-strict-encrypt docker image for platform linux/x86_64 ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-clickhouse-strict-encrypt test

@octavia-squidington-iii
Copy link
Collaborator

destination-starburst-galaxy test report (commit 8b74aa3a7f) - ❌

⏲️ Total pipeline duration: 17mn08s

Step Result
Validate airbyte-integrations/connectors/destination-starburst-galaxy/metadata.yaml ✅
Connector version semver check ✅
QA checks ❌
Build connector tar ✅
Build destination-starburst-galaxy docker image for platform linux/x86_64 ✅
./gradlew :airbyte-integrations:connectors:destination-starburst-galaxy:integrationTest ❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-starburst-galaxy test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jbfbell jbfbell jbfbell approved these changes

@edgao edgao edgao left review comments

@octavia-approvington octavia-approvington octavia-approvington approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues connectors/destination/bigquery
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@evantahler @octavia-squidington-iii @octavia-approvington @jbfbell @edgao