Proof of concept parallel source stream reading implementation for MySQL #26580

jdpgrailsdev · 2023-05-25T15:30:22Z

What

Proof of concept implementation that adds the ability to read streams in parallel in the MySQL source connector

How

Introduces a new method to the Source interface that returns the individual stream iterators instead of a composite iterator
Modifies the IntegrationRunner to use the new Source interface method. This new logic submits the consumption of records from each iterator in the list to a thread pool, passing each stream directly to the produceMessages method of the IntegrationRunner in parallel.
Modifies the existing MySQL source to implement the new Source interface method to return a list of iterators.
Refactor the stream status tracking to reduce duplicate code and make it possible for the concurrent processing to leverage it

Open Questions

How should the parallelism for the thread pool be chosen? Currently, the POC hard codes the max to 10, which is the current number of max database pool connections allowed in the connector code. Ideally, this value should probably be chosen or at least informed by the platform, as it knows how many cores are being assigned to the pod. It may also require input from the end-user, as they may want to control the number of concurrent connections allowed to their source.
What is the correct way to feature flag this? I opted to use the FeatureFlags class, which relies on environment variables. This will allow the platform to set the environment variable on the pod by interpreting the real feature flag client, which supports contexts (e.g. scoping to a connection or workspace). Open to suggestions on this front.
How does CDC work with this? If the MySql source is in CDC mode, how many iterators does it create? If it only creates one, then this will work just fine. If it creates more than one, we may need to add some logic to only select the concurrent execution when not in CDC mode.

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

❌ Sources (33)

Connector	Version	Changelog	Publish
`source-alloydb`	`2.0.28`	✅	✅
`source-alloydb-strict-encrypt`	`2.0.28`	🔵 (ignored)	🔵 (ignored)
`source-azure-blob-storage`	`0.1.0`	✅	✅
`source-bigquery`	`0.2.3`	✅	✅
`source-clickhouse`	`0.1.17`	✅	✅
`source-clickhouse-strict-encrypt`	`0.1.17`	🔵 (ignored)	🔵 (ignored)
`source-cockroachdb`	`0.1.22`	✅	✅
`source-cockroachdb-strict-encrypt`	`0.1.22`	🔵 (ignored)	🔵 (ignored)
`source-db2`	`0.1.20`	✅	✅
`source-db2-strict-encrypt`	❌ `0.1.19` (mismatch: `0.1.20`)	🔵 (ignored)	🔵 (ignored)
`source-dynamodb`	`0.1.2`	✅	✅
`source-e2e-test`	`2.1.4`	✅	✅
`source-e2e-test-cloud`	`2.1.4`	🔵 (ignored)	🔵 (ignored)
`source-elasticsearch`	`0.1.1`	✅	✅
`source-jdbc`	`0.3.5`	🔵 (ignored)	🔵 (ignored)
`source-kafka`	`0.2.3`	✅	✅
`source-mongodb-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-v2`	`0.1.19`	✅	✅
`source-mssql`	`1.0.19`	✅	✅
`source-mssql-strict-encrypt`	`1.0.19`	🔵 (ignored)	🔵 (ignored)
`source-mysql`	`2.0.25`	✅	✅
`source-mysql-strict-encrypt`	`2.0.25`	🔵 (ignored)	🔵 (ignored)
`source-oracle`	`0.3.25`	✅	✅
`source-oracle-strict-encrypt`	❌ `0.3.24` (mismatch: `0.3.25`)	🔵 (ignored)	🔵 (ignored)
`source-postgres`	`2.0.34`	✅	✅
`source-postgres-strict-encrypt`	`2.0.34`	🔵 (ignored)	🔵 (ignored)
`source-redshift`	`0.3.17`	✅	✅
`source-relational-db`	`0.3.1`	🔵 (ignored)	🔵 (ignored)
`source-scaffold-java-jdbc`	`0.1.0`	🔵 (ignored)	🔵 (ignored)
`source-sftp`	`0.1.2`	✅	✅
`source-snowflake`	`0.1.36`	✅	✅
`source-teradata`	`0.1.0`	✅	✅
`source-tidb`	`0.2.5`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (50)

Connector	Version	Changelog	Publish
`destination-azure-blob-storage`	`0.2.0`	✅	✅
`destination-bigquery`	`1.4.5`	✅	✅
`destination-bigquery-denormalized`	`1.4.1`	✅	✅
`destination-cassandra`	`0.1.4`	✅	✅
`destination-clickhouse`	`0.2.5`	✅	✅
`destination-clickhouse-strict-encrypt`	`0.2.5`	🔵 (ignored)	🔵 (ignored)
`destination-csv`	`1.0.0`	✅	✅
`destination-databricks`	`1.1.0`	✅	✅
`destination-dev-null`	`0.3.0`	🔵 (ignored)	🔵 (ignored)
`destination-doris`	`0.1.0`	✅	✅
`destination-dynamodb`	`0.1.7`	✅	✅
`destination-e2e-test`	`0.3.0`	✅	✅
`destination-elasticsearch`	`0.1.6`	✅	✅
`destination-elasticsearch-strict-encrypt`	`0.1.6`	🔵 (ignored)	🔵 (ignored)
`destination-exasol`	`0.1.1`	✅	✅
`destination-gcs`	`0.3.0`	✅	✅
`destination-iceberg`	`0.1.0`	✅	✅
`destination-kafka`	`0.1.10`	✅	✅
`destination-keen`	`0.2.4`	✅	✅
`destination-kinesis`	`0.1.5`	✅	✅
`destination-local-json`	`0.2.11`	✅	✅
`destination-mariadb-columnstore`	`0.1.7`	✅	✅
`destination-mongodb`	`0.1.9`	✅	✅
`destination-mongodb-strict-encrypt`	`0.1.9`	🔵 (ignored)	🔵 (ignored)
`destination-mqtt`	`0.1.3`	✅	✅
`destination-mssql`	`0.1.25`	✅	✅
`destination-mssql-strict-encrypt`	`0.1.25`	🔵 (ignored)	🔵 (ignored)
`destination-mysql`	`0.1.20`	✅	✅
`destination-mysql-strict-encrypt`	❌ `0.1.21` (mismatch: `0.1.20`)	🔵 (ignored)	🔵 (ignored)
`destination-oracle`	`0.1.19`	✅	✅
`destination-oracle-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`destination-postgres`	`0.3.27`	✅	✅
`destination-postgres-strict-encrypt`	`0.3.27`	🔵 (ignored)	🔵 (ignored)
`destination-pubsub`	`0.2.0`	✅	✅
`destination-pulsar`	`0.1.3`	✅	✅
`destination-r2`	`0.1.0`	✅	✅
`destination-redis`	`0.1.4`	✅	✅
`destination-redpanda`	`0.1.0`	✅	✅
`destination-redshift`	`0.4.9`	✅	✅
`destination-rockset`	`0.1.4`	✅	✅
`destination-s3`	`0.4.2`	✅	❌ (diff seed version)
`destination-s3-glue`	`0.1.7`	✅	✅
`destination-scylla`	`0.1.3`	✅	✅
`destination-selectdb`	`0.1.0`	✅	✅
`destination-snowflake`	`1.0.6`	✅	✅
`destination-starburst-galaxy`	`0.0.1`	✅	✅
`destination-teradata`	`0.1.1`	✅	✅
`destination-tidb`	`0.1.4`	✅	✅
`destination-vertica`	`0.1.0`	✅	✅
`destination-yugabytedb`	`0.1.1`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled.

benmoriceau · 2023-05-25T22:18:19Z

airbyte-commons/src/main/java/io/airbyte/commons/util/ParallelCompositeIterator.java

+     */
+    executorService = Executors.newFixedThreadPool(computeThreadPoolSize());
+    /*
+     * TODO Expose the max capacity as an advanced connector setting. This would allow the end user to


I agree with this, I wonder if it should be a configuration shared between the platform and the connector. My reasoning for sharing this config is that the platform orchestrator will also implement an internal buffer and will likely face the same issues than the connector if the records are too big. WDYT?

@benmoriceau Good point. I'm unclear at the moment as to whether or not an end user would even understand what this lever does. At a minimum, it should probably be in sync with the platform to ensure that we don't overrun it. That being said, the connector also has different memory restrictions than the platform, so the setting may not be equal to what the platform uses, but some smaller value or dynamic value that goes down on retry if the previous failure is due to an OOM. In any event, it needs to be configurable regardless of who sets the configuration.

benmoriceau · 2023-05-25T22:19:10Z

airbyte-commons/src/main/java/io/airbyte/commons/util/ParallelCompositeIterator.java

+
+      // Block while waiting for the next result to be available
+      final T result = internalQueue.take().data();
+      LOGGER.debug("Fetched next result {} from internal queue.", result);


NIT: this should not make it to prod, it activates the possibility of having client data in the logs.

benmoriceau

The approach LGTM

benmoriceau

The approach LGTM

subodh1810

I and @jdpgrailsdev went over the PR and we discussed two major things :

The internalQueue inside ParallelCompositeIterator class has a high probability of becoming a bottleneck. We have this design cause the read method today returns one iterator. But what if we change the read method to return a list of iterators and each iterator represents one single stream. That way we could start a thread for each stream and all it would do is consume records from the iterator and print it to the console. This way we dont need the queue and ParallelCompositeIterator. The entire logic would live in IntegrationRunner and we could put it inside a feature flag to easily commit this to master and test it out.
The exception handling also needs to make sure that an exception that happened while syncing a single stream is propagated properly downstream.

jdpgrailsdev · 2023-05-31T18:24:49Z

@subodh1810 @benmoriceau

The PR has been updated to introduce a different approach suggested by @subodh1810. The new approach moves the concurrency to the IntegrationRunner. This approach has a few benefits over the other one included in this PR (the "parallel" iterator):

It removes the need to have an internal queue between the actual read from each individual stream and the read -> publish to standard out logic. This is beneficial because it will be difficult to size the internal queue in such as way that it does not result in OOM's or becomes so limited that it doesn't help improve throughput.
It moves this logic to more of a general/CDK/platform-y location in the source. The IntegrationRunner is the glue that actually executes the read. This allows us to keep this logic out of the sources and to even move some other logic (like stream status publishing) to a more central location. It also brings it more inline with the Python CDK.
The addition of a new method to the Source interface allows us to easily turn on/implement this logic in other sources as we see fit without major impact to non-concurrent sources.

As mentioned above, I have left the ParallelCompositeItereator in the PR for now, both as a reference as you look at the new approach and just so we don't lose it in case we need to go back to it. I will clean up the PR and description once we are ok with the approach and ready to move this out of draft status.

cc: @davinchia

subodh1810

This makes sense, I just have concerns around the working of the method watchForOrphanThreads. I think its complicating the working of parallelism and may cause problems which would be difficult to debug.

jdpgrailsdev · 2023-06-02T12:30:05Z

This makes sense, I just have concerns around the working of the method watchForOrphanThreads. I think its complicating the working of parallelism and may cause problems which would be difficult to debug.

@subodh1810 Could you expound upon this? I do think there is a bug in the implementation -- I need to make sure that the executor service is stopped when the read is complete. That should take care of any threads created to do the concurrent read. I'll clean up the PR and get that change in there and we can continue the conversation about this once the logic to shut down the executor service is in place.

subodh1810

This makes sense, I am ready to approve. Just a compilation error and few changes that I am not sure belong in the PR.

subodh1810 · 2023-06-05T15:26:10Z

...elational-db/src/main/java/io/airbyte/integrations/source/relationaldb/AbstractDbSource.java

@@ -250,8 +250,7 @@ protected void estimateFullRefreshSyncSize(final Database database,
    /* no-op */
  }

-  private List<TableInfo<CommonField<DataType>>> discoverWithoutSystemTables(
-                                                                             final Database database)
+  private List<TableInfo<CommonField<DataType>>> discoverWithoutSystemTables(final Database database)


This needs to be protected, there is a compilation error I guess

Strange. I did not run into any issues building locally. I will make the suggested change to see if that helps.

subodh1810 · 2023-06-05T15:27:08Z

airbyte-commons/src/main/java/io/airbyte/commons/features/EnvVariableFeatureFlags.java

  public static final String FIELD_SELECTION_WORKSPACES = "FIELD_SELECTION_WORKSPACES";
-
-  @Override
-  public boolean autoDisablesFailingConnections() {


Why these changes?

They are no longer used. As I was in there adding a new flag, I figured it would be worth cleaning up the unused ones.

subodh1810 · 2023-06-05T15:28:55Z

Can you also run the /test command for Postgres and MySQL just to make sure that the changes are not breaking existing flow.

octavia-squidington-iii · 2023-08-04T14:25:37Z

destination-pubsub test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 19.17s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pubsub test

octavia-squidington-iii · 2023-08-04T14:26:09Z

destination-dev-null test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 16.81s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-dev-null test

octavia-squidington-iii · 2023-08-04T14:26:46Z

destination-bigquery-denormalized test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 22.37s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

octavia-squidington-iii · 2023-08-04T14:27:25Z

source-postgres test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 23.73s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

octavia-squidington-iii · 2023-08-04T14:28:00Z

source-mongodb-strict-encrypt test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 20.26s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

octavia-squidington-iii · 2023-08-04T14:28:39Z

destination-redshift test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 23.59s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

octavia-squidington-iii · 2023-08-04T14:29:18Z

source-snowflake test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 23.39s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-snowflake test

octavia-squidington-iii · 2023-08-04T14:29:52Z

destination-iceberg test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 18.25s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-iceberg test

octavia-squidington-iii · 2023-08-04T14:30:30Z

source-dynamodb test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 22.11s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-dynamodb test

octavia-squidington-iii · 2023-08-04T14:31:10Z

destination-s3 test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 24.49s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-s3 test

octavia-squidington-iii · 2023-08-04T14:31:46Z

destination-mssql test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 19.53s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mssql test

octavia-squidington-iii · 2023-08-04T14:32:26Z

destination-postgres test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 24.11s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-postgres test

octavia-squidington-iii · 2023-08-04T14:33:04Z

destination-e2e-test test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 20.71s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-e2e-test test

octavia-squidington-iii · 2023-08-04T14:33:43Z

destination-csv test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 21.24s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-csv test

octavia-squidington-iii · 2023-08-04T14:34:20Z

destination-vertica test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 20.99s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-vertica test

octavia-squidington-iii · 2023-08-04T14:35:00Z

destination-doris test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 21.69s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-doris test

octavia-squidington-iii · 2023-08-04T14:35:45Z

destination-gcs test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 28.03s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-gcs test

octavia-squidington-iii · 2023-08-04T14:36:28Z

source-redshift test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 26.58s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-redshift test

octavia-squidington-iii · 2023-08-04T14:37:11Z

destination-scylla test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 23.05s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-scylla test

octavia-squidington-iii · 2023-08-04T14:37:53Z

source-clickhouse test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 23.59s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-clickhouse test

octavia-squidington-iii · 2023-08-04T14:38:37Z

source-sftp test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 23.54s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-sftp test

octavia-squidington-iii · 2023-08-04T14:39:23Z

destination-tidb test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 25.61s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-tidb test

octavia-squidington-iii · 2023-08-04T14:40:10Z

source-elasticsearch test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 27.34s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-elasticsearch test

octavia-squidington-iii · 2023-08-04T14:40:57Z

destination-yugabytedb test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 27.46s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-yugabytedb test

octavia-squidington-iii · 2023-08-04T14:41:48Z

source-alloydb-strict-encrypt test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 28.45s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb-strict-encrypt test

octavia-squidington-iii · 2023-08-04T14:42:35Z

source-cockroachdb test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 27.49s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-cockroachdb test

octavia-squidington-iii · 2023-08-04T14:43:34Z

destination-databricks test report (commit `3788088485`) - ❌

⏲️ Total pipeline duration: 40.20s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-databricks test

…SQL (#26580) * Proof of concept parallel source stream reading implementation for MySQL * Automated Change * Add read method that supports concurrent execution to Source interface * Remove parallel iterator * Ensure that executor service is stopped * Automated Commit - Format and Process Resources Changes * Expose method to fix compilation issue * Use concurrent map to avoid access issues * Automated Commit - Format and Process Resources Changes * Ensure concurrent streams finish before closing source * Fix compile issue * Formatting * Exclude concurrent stream threads from orphan thread watcher * Automated Commit - Format and Process Resources Changes * Refactor orphaned thread logic to account for concurrent execution * PR feedback * Implement readStreams in wrapper source * Automated Commit - Format and Process Resources Changes * Add readStream override * Automated Commit - Format and Process Resources Changes * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * Debug logging * Reduce logging level * Replace synchronized calls to System.out.println when concurrent * Close consumer * Flush before close * Automated Commit - Format and Process Resources Changes * Remove charset * Use ASCII and flush periodically for parallel streams * Test performance harness patch * Automated Commit - Format and Process Resources Changes * Cleanup * Logging to identify concurrent read enabled * Mark parameter as final --------- Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com>

* Add everything for BQ but migrate, refactor interface after practical work * Make new default methods, refactor to single implemented method * MigrationInterface and BQ impl created * Trying to integrate with standard inserts * remove unnecessary NameAndNamespacePair class * Shimmed in * Java Docs * Initial Testing Setup * Tests! * Move Migrator into TyperDeduper * Functional Migration * Add Integration Test * Pr updates * bump version * bump version * version bump * Update to airbyte-ci-internal (#29026) * 🐛 Source Github, Instagram, Zendesk-support, Zendesk-talk: fix CAT tests fail on `spec` (#28910) * connectors-ci: better modified connectors detection logic (#28855) * connectors-ci: report path should always start with `airbyte-ci/` (#29030) * make report path always start with airbyte-ci * revert report path in orchestrator * add more test cases * bump version * Updated docs (#29019) * CDK: Embedded reader utils (#28873) * relax pydantic dep * Automated Commit - Format and Process Resources Changes * wip * wrap up base integration * add init file * introduce CDK runner and improve error message * make state param optional * update protocol models * review comments * always run incremental if possible * fix --------- Co-authored-by: flash1293 <flash1293@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * 🚨🚨 Low code CDK: Decouple SimpleRetriever and HttpStream (#28657) * fix tests * format * review comments * Automated Commit - Formatting Changes * review comments * review comments * review comments * log all messages * log all message * review comments * review comments * Automated Commit - Formatting Changes * add comment --------- Co-authored-by: flash1293 <flash1293@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * 🐛 Source Github, Instagram, Zendesk Support / Talk - revert `spec` changes and improve (#29031) * Source oauth0: new streams and fix incremental (#29001) * Add new streams Organizations,OrganizationMembers,OrganizationMemberRoles * relax schema definition to allow additional fields * Bump image tag version * revert some changes to the old schemas * Format python so gradle can pass * update incremental * remove unused print * fix unit test --------- Co-authored-by: Vasilis Gavriilidis <vasilis.gavriilidis@orfium.com> * 🐛 Source Mongo: Fix failing acceptance tests (#28816) * Fix failing acceptance tests * Fix failing strict acceptance tests * Source-Greenhouse: Fix unit tests for new CDK version (#28969) Fix unit tests * Add CSV options to the CSV parser (#28491) * remove invalid legacy option * remove unused option * the tests pass but this is quite messy * very slight clean up * Add skip options to csv format * fix some of the typing issues * fixme comment * remove extra log message * fix typing issues * skip before header * skip after header * format * add another test * Automated Commit - Formatting Changes * auto generate column names * delete dead code * update title and description * true and false values * Update the tests * Add comment * missing test * rename * update expected spec * move to method * Update comment * fix typo * remove unused import * Add a comment * None records do not pass the WaitForDiscoverPolicy * format * remove second branch to ensure we always go through the same processing * Raise an exception if the record is None * reset * Update tests * handle unquoted newlines * Automated Commit - Formatting Changes * Update test case so the quoting is explicit * Update comment * Automated Commit - Formatting Changes * Fail validation if skipping rows before header and header is autogenerated * always fail if a record cannot be parsed * format * set write line_no in error message * remove none check * Automated Commit - Formatting Changes * enable autogenerate test * remove duplicate test * missing unit tests * Update * remove branching * remove unused none check * Update tests * remove branching * format * extract to function * comment * missing type * type annotation * use set * Document that the strings are case-sensitive * public -> private * add unit test * newline --------- Co-authored-by: girarda <girarda@users.noreply.github.com> * Dagster: Add sentry logging (#28822) * Add sentry * add sentry decorator * Add traces * Use sentry trace * Improve duplicate logging * Add comments * DNC * Fix up issues * Move to scopes * Remove breadcrumb * Update lock * ✨Source Shortio: Migrate Python CDK to Low-code CDK (#28950) * Migrate Shortio to Low-Code * Update abnormal state * Format * Update Docs * Fix metadata.yaml * Add pagination * Add incremental sync * add incremental parameters * update metadata * rollback update version * release date --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> * Update to new verbiage (#29051) * [skip ci] Metadata: Remove leading underscore (#29024) * DNC * Add test models * Add model test * Remove underscore from metadata files * Regenerate models * Add test to check for key transformation * Allow additional fields on metadata * Delete transform * Proof of concept parallel source stream reading implementation for MySQL (#26580) * Proof of concept parallel source stream reading implementation for MySQL * Automated Change * Add read method that supports concurrent execution to Source interface * Remove parallel iterator * Ensure that executor service is stopped * Automated Commit - Format and Process Resources Changes * Expose method to fix compilation issue * Use concurrent map to avoid access issues * Automated Commit - Format and Process Resources Changes * Ensure concurrent streams finish before closing source * Fix compile issue * Formatting * Exclude concurrent stream threads from orphan thread watcher * Automated Commit - Format and Process Resources Changes * Refactor orphaned thread logic to account for concurrent execution * PR feedback * Implement readStreams in wrapper source * Automated Commit - Format and Process Resources Changes * Add readStream override * Automated Commit - Format and Process Resources Changes * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * Debug logging * Reduce logging level * Replace synchronized calls to System.out.println when concurrent * Close consumer * Flush before close * Automated Commit - Format and Process Resources Changes * Remove charset * Use ASCII and flush periodically for parallel streams * Test performance harness patch * Automated Commit - Format and Process Resources Changes * Cleanup * Logging to identify concurrent read enabled * Mark parameter as final --------- Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> * connectors-ci: disable dependency scanning (#29033) * updates (#29059) * Metadata: skip breaking change validation on prerelease (#29017) * skip breaking change validation * Move ValidatorOpts higher in call * Add prerelease test * Fix test * ✨ Source MongoDB Internal POC: Generate Test Data (#29049) * Add script to generate test data * Fix prose * Update credentials example * PR feedback * Bump Airbyte version from 0.50.12 to 0.50.13 * Bump versions for mssql strict-encrypt (#28964) * Bump versions for mssql strict-encrypt * Fix failing test * Fix failing test * 🎨 Improve replication method selection UX (#28882) * update replication method in MySQL source * bump version * update expected specs * update registries * bump strict encrypt version * make password always_show * change url * update registries * 🐛 Avoid writing records to log (#29047) * Avoid writing records to log * Update version * Rollout ctid cdc (#28708) * source-postgres: enable ctid+cdc implementation * 100% ctid rollout for cdc * remove CtidFeatureFlags * fix CdcPostgresSourceAcceptanceTest * Bump versions and release notes * Fix compilation error due to previous merge --------- Co-authored-by: subodh <subodh1810@gmail.com> * connectors-ci: fix `unhashable type 'set'` (#29064) * Add Slack Alert lifecycle to Dagster for Metadata publish (#28759) * DNC * Add slack lifecycle logging * Update to use slack * Update slack to use resource and bot * Improve markdown * Improve log * Add sensor logging * Extend sensor time * merge conflict * PR Refactoring * Make the tests work * remove unnecessary classes, pr feedback * more merging * Update airbyte-integrations/bases/base-typing-deduping-test/src/main/java/io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java Co-authored-by: Edward Gao <edward.gao@airbyte.io> * snowflake updates --------- Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com> Co-authored-by: Joe Reuter <joe@airbyte.io> Co-authored-by: flash1293 <flash1293@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Vasilis Gavriilidis <vasilis.gavriilidis@orfium.com> Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io> Co-authored-by: Alexandre Girard <alexandre@airbyte.io> Co-authored-by: girarda <girarda@users.noreply.github.com> Co-authored-by: btkcodedev <btk.codedev@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com> Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com> Co-authored-by: terencecho <terencecho@users.noreply.github.com> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> Co-authored-by: subodh <subodh1810@gmail.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io>

* Add everything for BQ but migrate, refactor interface after practical work * Make new default methods, refactor to single implemented method * MigrationInterface and BQ impl created * Trying to integrate with standard inserts * remove unnecessary NameAndNamespacePair class * Shimmed in * Java Docs * Initial Testing Setup * Tests! * Move Migrator into TyperDeduper * Functional Migration * Add Integration Test * Pr updates * bump version * bump version * version bump * Update to airbyte-ci-internal (airbytehq#29026) * 🐛 Source Github, Instagram, Zendesk-support, Zendesk-talk: fix CAT tests fail on `spec` (airbytehq#28910) * connectors-ci: better modified connectors detection logic (airbytehq#28855) * connectors-ci: report path should always start with `airbyte-ci/` (airbytehq#29030) * make report path always start with airbyte-ci * revert report path in orchestrator * add more test cases * bump version * Updated docs (airbytehq#29019) * CDK: Embedded reader utils (airbytehq#28873) * relax pydantic dep * Automated Commit - Format and Process Resources Changes * wip * wrap up base integration * add init file * introduce CDK runner and improve error message * make state param optional * update protocol models * review comments * always run incremental if possible * fix --------- Co-authored-by: flash1293 <flash1293@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * 🚨🚨 Low code CDK: Decouple SimpleRetriever and HttpStream (airbytehq#28657) * fix tests * format * review comments * Automated Commit - Formatting Changes * review comments * review comments * review comments * log all messages * log all message * review comments * review comments * Automated Commit - Formatting Changes * add comment --------- Co-authored-by: flash1293 <flash1293@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * 🐛 Source Github, Instagram, Zendesk Support / Talk - revert `spec` changes and improve (airbytehq#29031) * Source oauth0: new streams and fix incremental (airbytehq#29001) * Add new streams Organizations,OrganizationMembers,OrganizationMemberRoles * relax schema definition to allow additional fields * Bump image tag version * revert some changes to the old schemas * Format python so gradle can pass * update incremental * remove unused print * fix unit test --------- Co-authored-by: Vasilis Gavriilidis <vasilis.gavriilidis@orfium.com> * 🐛 Source Mongo: Fix failing acceptance tests (airbytehq#28816) * Fix failing acceptance tests * Fix failing strict acceptance tests * Source-Greenhouse: Fix unit tests for new CDK version (airbytehq#28969) Fix unit tests * Add CSV options to the CSV parser (airbytehq#28491) * remove invalid legacy option * remove unused option * the tests pass but this is quite messy * very slight clean up * Add skip options to csv format * fix some of the typing issues * fixme comment * remove extra log message * fix typing issues * skip before header * skip after header * format * add another test * Automated Commit - Formatting Changes * auto generate column names * delete dead code * update title and description * true and false values * Update the tests * Add comment * missing test * rename * update expected spec * move to method * Update comment * fix typo * remove unused import * Add a comment * None records do not pass the WaitForDiscoverPolicy * format * remove second branch to ensure we always go through the same processing * Raise an exception if the record is None * reset * Update tests * handle unquoted newlines * Automated Commit - Formatting Changes * Update test case so the quoting is explicit * Update comment * Automated Commit - Formatting Changes * Fail validation if skipping rows before header and header is autogenerated * always fail if a record cannot be parsed * format * set write line_no in error message * remove none check * Automated Commit - Formatting Changes * enable autogenerate test * remove duplicate test * missing unit tests * Update * remove branching * remove unused none check * Update tests * remove branching * format * extract to function * comment * missing type * type annotation * use set * Document that the strings are case-sensitive * public -> private * add unit test * newline --------- Co-authored-by: girarda <girarda@users.noreply.github.com> * Dagster: Add sentry logging (airbytehq#28822) * Add sentry * add sentry decorator * Add traces * Use sentry trace * Improve duplicate logging * Add comments * DNC * Fix up issues * Move to scopes * Remove breadcrumb * Update lock * ✨Source Shortio: Migrate Python CDK to Low-code CDK (airbytehq#28950) * Migrate Shortio to Low-Code * Update abnormal state * Format * Update Docs * Fix metadata.yaml * Add pagination * Add incremental sync * add incremental parameters * update metadata * rollback update version * release date --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> * Update to new verbiage (airbytehq#29051) * [skip ci] Metadata: Remove leading underscore (airbytehq#29024) * DNC * Add test models * Add model test * Remove underscore from metadata files * Regenerate models * Add test to check for key transformation * Allow additional fields on metadata * Delete transform * Proof of concept parallel source stream reading implementation for MySQL (airbytehq#26580) * Proof of concept parallel source stream reading implementation for MySQL * Automated Change * Add read method that supports concurrent execution to Source interface * Remove parallel iterator * Ensure that executor service is stopped * Automated Commit - Format and Process Resources Changes * Expose method to fix compilation issue * Use concurrent map to avoid access issues * Automated Commit - Format and Process Resources Changes * Ensure concurrent streams finish before closing source * Fix compile issue * Formatting * Exclude concurrent stream threads from orphan thread watcher * Automated Commit - Format and Process Resources Changes * Refactor orphaned thread logic to account for concurrent execution * PR feedback * Implement readStreams in wrapper source * Automated Commit - Format and Process Resources Changes * Add readStream override * Automated Commit - Format and Process Resources Changes * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * Debug logging * Reduce logging level * Replace synchronized calls to System.out.println when concurrent * Close consumer * Flush before close * Automated Commit - Format and Process Resources Changes * Remove charset * Use ASCII and flush periodically for parallel streams * Test performance harness patch * Automated Commit - Format and Process Resources Changes * Cleanup * Logging to identify concurrent read enabled * Mark parameter as final --------- Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> * connectors-ci: disable dependency scanning (airbytehq#29033) * updates (airbytehq#29059) * Metadata: skip breaking change validation on prerelease (airbytehq#29017) * skip breaking change validation * Move ValidatorOpts higher in call * Add prerelease test * Fix test * ✨ Source MongoDB Internal POC: Generate Test Data (airbytehq#29049) * Add script to generate test data * Fix prose * Update credentials example * PR feedback * Bump Airbyte version from 0.50.12 to 0.50.13 * Bump versions for mssql strict-encrypt (airbytehq#28964) * Bump versions for mssql strict-encrypt * Fix failing test * Fix failing test * 🎨 Improve replication method selection UX (airbytehq#28882) * update replication method in MySQL source * bump version * update expected specs * update registries * bump strict encrypt version * make password always_show * change url * update registries * 🐛 Avoid writing records to log (airbytehq#29047) * Avoid writing records to log * Update version * Rollout ctid cdc (airbytehq#28708) * source-postgres: enable ctid+cdc implementation * 100% ctid rollout for cdc * remove CtidFeatureFlags * fix CdcPostgresSourceAcceptanceTest * Bump versions and release notes * Fix compilation error due to previous merge --------- Co-authored-by: subodh <subodh1810@gmail.com> * connectors-ci: fix `unhashable type 'set'` (airbytehq#29064) * Add Slack Alert lifecycle to Dagster for Metadata publish (airbytehq#28759) * DNC * Add slack lifecycle logging * Update to use slack * Update slack to use resource and bot * Improve markdown * Improve log * Add sensor logging * Extend sensor time * merge conflict * PR Refactoring * Make the tests work * remove unnecessary classes, pr feedback * more merging * Update airbyte-integrations/bases/base-typing-deduping-test/src/main/java/io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java Co-authored-by: Edward Gao <edward.gao@airbyte.io> * snowflake updates --------- Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com> Co-authored-by: Joe Reuter <joe@airbyte.io> Co-authored-by: flash1293 <flash1293@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Vasilis Gavriilidis <vasilis.gavriilidis@orfium.com> Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io> Co-authored-by: Alexandre Girard <alexandre@airbyte.io> Co-authored-by: girarda <girarda@users.noreply.github.com> Co-authored-by: btkcodedev <btk.codedev@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com> Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com> Co-authored-by: terencecho <terencecho@users.noreply.github.com> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> Co-authored-by: subodh <subodh1810@gmail.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io>

Proof of concept parallel source stream reading implementation for MySQL

8e2b78a

jdpgrailsdev requested review from benmoriceau, davinchia and subodh1810 May 25, 2023 15:30

octavia-squidington-iii added area/connectors Connector related issues connectors/source/mysql connectors/source/relational-db labels May 25, 2023

Automated Change

ffd4cdb

benmoriceau reviewed May 25, 2023

View reviewed changes

Merge branch 'master' into jonathan/mysql-parallel-source-poc

32b1784

subodh1810 reviewed May 30, 2023

View reviewed changes

Add read method that supports concurrent execution to Source interface

a802edb

subodh1810 reviewed Jun 2, 2023

View reviewed changes

jdpgrailsdev added 2 commits June 2, 2023 08:39

Remove parallel iterator

c1f2c47

Ensure that executor service is stopped

cd28028

jdpgrailsdev force-pushed the jonathan/mysql-parallel-source-poc branch from 0a7414d to cd28028 Compare June 2, 2023 12:40

Merge branch 'master' into jonathan/mysql-parallel-source-poc

9716328

jdpgrailsdev marked this pull request as ready for review June 2, 2023 12:51

jdpgrailsdev requested review from a team as code owners June 2, 2023 12:51

Automated Commit - Format and Process Resources Changes

a2f7caa

subodh1810 reviewed Jun 5, 2023

View reviewed changes

Expose method to fix compilation issue

4cdd325

Proof of concept parallel source stream reading implementation for MySQL #26580

Proof of concept parallel source stream reading implementation for MySQL #26580

Conversation

jdpgrailsdev commented May 25, 2023 • edited Loading

What

How

Open Questions

Recommended reading order

github-actions bot commented May 25, 2023 • edited by jdpgrailsdev Loading

Before Merging a Connector Pull Request

github-actions bot commented May 25, 2023 • edited Loading

Affected Connector Report

❌ Sources (33)

❌ Destinations (50)

✅ Other Modules (0)

Actionable Items

benmoriceau May 25, 2023

Choose a reason for hiding this comment

jdpgrailsdev May 26, 2023

Choose a reason for hiding this comment

benmoriceau May 25, 2023

Choose a reason for hiding this comment

benmoriceau left a comment

Choose a reason for hiding this comment

benmoriceau left a comment

Choose a reason for hiding this comment

subodh1810 left a comment

Choose a reason for hiding this comment

jdpgrailsdev commented May 31, 2023

subodh1810 left a comment

Choose a reason for hiding this comment

jdpgrailsdev commented Jun 2, 2023

subodh1810 left a comment

Choose a reason for hiding this comment

subodh1810 Jun 5, 2023

Choose a reason for hiding this comment

jdpgrailsdev Jun 5, 2023

Choose a reason for hiding this comment

subodh1810 Jun 5, 2023

Choose a reason for hiding this comment

jdpgrailsdev Jun 5, 2023

Choose a reason for hiding this comment

subodh1810 commented Jun 5, 2023

octavia-squidington-iii commented Aug 4, 2023

destination-pubsub test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-dev-null test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-bigquery-denormalized test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

source-postgres test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

source-mongodb-strict-encrypt test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-redshift test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

source-snowflake test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-iceberg test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

source-dynamodb test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-s3 test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-mssql test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-postgres test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-e2e-test test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-csv test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-vertica test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-doris test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

destination-gcs test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

source-redshift test report (commit 3788088485) - ❌

octavia-squidington-iii commented Aug 4, 2023

jdpgrailsdev commented May 25, 2023 •

edited

Loading

github-actions bot commented May 25, 2023 •

edited by jdpgrailsdev

Loading

github-actions bot commented May 25, 2023 •

edited

Loading

destination-pubsub test report (commit `3788088485`) - ❌

destination-dev-null test report (commit `3788088485`) - ❌

destination-bigquery-denormalized test report (commit `3788088485`) - ❌

source-postgres test report (commit `3788088485`) - ❌

source-mongodb-strict-encrypt test report (commit `3788088485`) - ❌

destination-redshift test report (commit `3788088485`) - ❌

source-snowflake test report (commit `3788088485`) - ❌

destination-iceberg test report (commit `3788088485`) - ❌

source-dynamodb test report (commit `3788088485`) - ❌

destination-s3 test report (commit `3788088485`) - ❌

destination-mssql test report (commit `3788088485`) - ❌

destination-postgres test report (commit `3788088485`) - ❌

destination-e2e-test test report (commit `3788088485`) - ❌

destination-csv test report (commit `3788088485`) - ❌

destination-vertica test report (commit `3788088485`) - ❌

destination-doris test report (commit `3788088485`) - ❌

destination-gcs test report (commit `3788088485`) - ❌

source-redshift test report (commit `3788088485`) - ❌

destination-scylla test report (commit `3788088485`) - ❌

source-clickhouse test report (commit `3788088485`) - ❌

source-sftp test report (commit `3788088485`) - ❌

destination-tidb test report (commit `3788088485`) - ❌

source-elasticsearch test report (commit `3788088485`) - ❌

destination-yugabytedb test report (commit `3788088485`) - ❌

source-alloydb-strict-encrypt test report (commit `3788088485`) - ❌

source-cockroachdb test report (commit `3788088485`) - ❌

destination-databricks test report (commit `3788088485`) - ❌