✨ Source MongoDB Internal POC: CDC State Handling #29763

jdpgrailsdev · 2023-08-23T17:43:21Z

What

Add state handling/management for both initial snapshots and incremental syncs as part of CDC for MongoDB

How

Implement the CdcStateHandler interface so that state is updated as part of the CDC incremental sync
Add a state manager based on the relational database state management code that can be used to keep track of state as both initial snapshots are performed and as part of incremental syncs
Refactor existing code to use the state manager for state tracking

source-mssql test report (commit `3ab0fe24d6`) - ❌

⏲️ Total pipeline duration: 18mn06s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mssql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mssql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

github-actions · 2023-08-23T18:09:37Z

Coverage report for source-postgres

File	Coverage [100%]	🍏
PostgresCdcConnectorMetadataInjector.java	100%	🍏

Total Project Coverage	70.2%	🍏

github-actions · 2023-08-23T18:37:20Z

source-mysql test report (commit `3ab0fe24d6`) - ❌

⏲️ Total pipeline duration: 33mn59s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-08-23T18:45:02Z

source-mongodb-internal-poc test report (commit `3ab0fe24d6`) - ❌

⏲️ Total pipeline duration: 07mn32s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mongodb-internal-poc docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mongodb-internal-poc/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-internal-poc test

github-actions · 2023-08-23T19:10:02Z

source-postgres test report (commit `3ab0fe24d6`) - ❌

⏲️ Total pipeline duration: 24mn48s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	❌
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-08-23T19:20:20Z

source-mongodb-internal-poc test report (commit `2f2fe533eb`) - ❌

⏲️ Total pipeline duration: 08mn28s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mongodb-internal-poc docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mongodb-internal-poc/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-internal-poc test

github-actions · 2023-08-23T19:37:24Z

source-mysql test report (commit `2f2fe533eb`) - ❌

⏲️ Total pipeline duration: 16mn52s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-08-23T20:12:06Z

source-postgres test report (commit `2f2fe533eb`) - ❌

⏲️ Total pipeline duration: 34mn31s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	❌
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-08-23T20:30:53Z

source-mssql test report (commit `2f2fe533eb`) - ❌

⏲️ Total pipeline duration: 18mn35s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mssql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mssql/metadata.yaml	✅
Connector version semver check	❌
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

github-actions · 2023-08-23T20:42:44Z

source-mongodb-internal-poc test report (commit `9317ccfd6d`) - ❌

⏲️ Total pipeline duration: 09mn20s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mongodb-internal-poc docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mongodb-internal-poc/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-internal-poc test

github-actions · 2023-08-23T21:19:44Z

source-mssql test report (commit `9317ccfd6d`) - ❌

⏲️ Total pipeline duration: 36mn50s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mssql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mssql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

github-actions · 2023-08-24T17:01:30Z

source-mongodb-internal-poc test report (commit `9517f34860`) - ❌

⏲️ Total pipeline duration: 09mn15s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mongodb-internal-poc docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mongodb-internal-poc/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-internal-poc test

github-actions · 2023-08-24T19:59:02Z

source-mysql test report (commit `cbf5f095e8`) - ❌

⏲️ Total pipeline duration: 17mn44s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-08-24T20:02:46Z

source-postgres test report (commit `cbf5f095e8`) - ❌

⏲️ Total pipeline duration: 03mn34s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

jpefaur

I think I understand what's going on here. I'm going to read it again tomorrow morning, but feel free to merge

subodh1810

Few questions/concerns :

Need to understand if swallowing exception in the hasNext method of MongoDbStateIterator is the right thing to do?
How would we identify that initial sync for a given stream is complete. I dont see any special state emitted for streams that complete initial sync to denote now we just need to process the change streams for these streams. Which leads to the next question is that I dont see any specific logic to identify which streams only need to go through the initial sync. Right now all the streams present in the catalog go through the intial sync all the time.
Other DBs emit state at a frequency of every 15 minutes or once more than 10K records are emiited. May be we could do the same thing?

subodh1810 · 2023-08-25T11:20:58Z

...src/main/java/io/airbyte/integrations/source/mongodb/internal/state/MongoDbStateManager.java

+      if (!stateMessages.isEmpty()) {
+        if (stateMessages.size() == 1) {
+          final AirbyteStateMessage stateMessage = stateMessages.get(0);
+          stateManager.updateCdcState(Jsons.object(stateMessage.getGlobal().getSharedState(), MongoDbCdcState.class));


Isn't the CDC state supposed to be the debezium state? How will it be de-serialised to MongoDbCdcState

The MongoDbCdcState holds the data extracted from the Debezium offset map. There is code to handle the translation to/from the offset map and MongoDbCdcState that is not in this PR, but will be in the PR that integrates all the bits together.

...-poc/src/main/java/io/airbyte/integrations/source/mongodb/internal/MongoDbStateIterator.java

jdpgrailsdev · 2023-08-25T14:24:02Z

Few questions/concerns :

Need to understand if swallowing exception in the hasNext method of MongoDbStateIterator is the right thing to do?

How would we identify that initial sync for a given stream is complete. I dont see any special state emitted for streams that complete initial sync to denote now we just need to process the change streams for these streams. Which leads to the next question is that I dont see any specific logic to identify which streams only need to go through the initial sync. Right now all the streams present in the catalog go through the intial sync all the time.

Other DBs emit state at a frequency of every 15 minutes or once more than 10K records are emiited. May be we could do the same thing?

@subodh1810 I think the answer to #1 is that it is the correct thing to do, but I will defer to @colesnodgrass as he wrote that code. As for #2 and #3, those are out of scope of this PR. The intent here is to just set up the initial structure. We will need to handle those two cases when we integrate this with the iterator, which itself will probably need to move/change to be part of the code that sets up the iterators for the CDC sync given the state.

…-handler

github-actions · 2023-08-25T14:51:43Z

source-mongodb-internal-poc test report (commit `af7feba553`) - ❌

⏲️ Total pipeline duration: 09mn16s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mongodb-internal-poc docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mongodb-internal-poc/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-internal-poc test

…-handler

github-actions · 2023-08-25T15:31:20Z

source-mongodb-internal-poc test report (commit `291f6ff50c`) - ❌

⏲️ Total pipeline duration: 09mn20s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mongodb-internal-poc docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mongodb-internal-poc/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-internal-poc test

github-actions · 2023-08-25T15:42:12Z

source-mongodb-internal-poc test report (commit `291f6ff50c`) - ❌

⏲️ Total pipeline duration: 08mn37s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mongodb-internal-poc docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mongodb-internal-poc/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-internal-poc test

…-handler

github-actions · 2023-08-25T17:05:28Z

source-mysql test report (commit `28e4dc0380`) - ❌

⏲️ Total pipeline duration: 35mn49s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-08-25T17:13:15Z

source-mongodb-internal-poc test report (commit `28e4dc0380`) - ❌

⏲️ Total pipeline duration: 07mn37s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mongodb-internal-poc docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mongodb-internal-poc/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-internal-poc test

github-actions · 2023-08-25T17:47:04Z

source-postgres test report (commit `28e4dc0380`) - ❌

⏲️ Total pipeline duration: 33mn38s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-08-25T17:47:20Z

source-mssql test report (commit `28e4dc0380`) - ❌

⏲️ Total pipeline duration: 6.52s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

subodh1810

Looks good. Again lets make sure that the tests for all the other CDC connectors are green.

Also as discussed over chat I pushed a commit to disable the following methods

  @Override
  public AirbyteMessage saveStateAfterCompletionOfSnapshotOfNewStreams() {
    throw new RuntimeException("Debezium is not used to carry out the snapshot of tables.");
  }

  @Override
  public boolean isCdcCheckpointEnabled() {
    return false;
  }

The method saveStateAfterCompletionOfSnapshotOfNewStreams is not required cause we wont be using Debezium for initial snapshot of streams.

The method isCdcCheckpointEnabled requires the following methods in CdcTargetPosition to be implemented as well. So an item on the TODO list to implement these methods in the future PR and then we can set the isCdcCheckpointEnabled to true.

default boolean isEventAheadOffset(final Map<String, String> offset, final ChangeEventWithMetadata event) {
   return false;
 }

and

  default boolean isSameOffset(final Map<String, String> offsetA, final Map<String, String> offsetB) {
    return true;
  }

subodh1810 · 2023-08-25T18:06:33Z

...io/airbyte/integrations/source/mongodb/internal/cdc/MongoDbCdcConnectorMetadataInjector.java

 import java.util.concurrent.atomic.AtomicLong;
+import org.bson.BsonTimestamp;


Not required right?

github-actions · 2023-08-25T18:17:10Z

source-mongodb-internal-poc test report (commit `971af014e2`) - ❌

⏲️ Total pipeline duration: 09mn23s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mongodb-internal-poc docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mongodb-internal-poc/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-internal-poc test

github-actions · 2023-08-25T18:17:25Z

source-mssql test report (commit `971af014e2`) - ❌

⏲️ Total pipeline duration: 6.25s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

github-actions · 2023-08-25T18:34:37Z

source-mysql test report (commit `971af014e2`) - ❌

⏲️ Total pipeline duration: 17mn01s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

github-actions · 2023-08-25T18:46:00Z

source-postgres test report (commit `971af014e2`) - ❌

⏲️ Total pipeline duration: 11mn12s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

…-handler

github-actions · 2023-08-28T13:59:01Z

source-mongodb-internal-poc test report (commit `9a13b5fae7`) - ❌

⏲️ Total pipeline duration: 09mn06s

Step	Result
Java Connector Unit Tests	❌
Build connector tar	✅
Build source-mongodb-internal-poc docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-mongodb-internal-poc/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-internal-poc test

jdpgrailsdev added 2 commits August 23, 2023 13:25

Add state management and handling for CDC integration

f3f0ead

Formatting

3ab0fe2

jdpgrailsdev requested review from colesnodgrass, jpefaur and subodh1810 August 23, 2023 17:43

jdpgrailsdev requested a review from a team as a code owner August 23, 2023 17:43

octavia-squidington-iii added area/connectors Connector related issues connectors/source/mssql connectors/source/mysql connectors/source/postgres connectors/source/mongodb-internal-poc labels Aug 23, 2023

jdpgrailsdev mentioned this pull request Aug 23, 2023

✨ Source MongoDB Internal POC: Use global state for CDC #29678

Closed

jdpgrailsdev and others added 2 commits August 23, 2023 13:44

Fix typo

4b5552f

Merge branch 'master' into jonathan/cdc-state-handler

2f2fe53

jdpgrailsdev added 2 commits August 23, 2023 15:38

Merge branch 'master' into jonathan/cdc-state-handler

f67aa27

Merge branch 'master' into jonathan/cdc-state-handler

9317ccf

Merge branch 'master' into jonathan/cdc-state-handler

9517f34

jpefaur approved these changes Aug 25, 2023

View reviewed changes

subodh1810 requested changes Aug 25, 2023

View reviewed changes

jdpgrailsdev changed the base branch from master to jonathan/cdc-metadata-injector August 25, 2023 12:51

Merge branch 'jonathan/cdc-metadata-injector' into jonathan/cdc-state…

af7feba

…-handler

jdpgrailsdev and others added 2 commits August 25, 2023 11:17

Merge branch 'jonathan/cdc-metadata-injector' into jonathan/cdc-state…

a8b2455

…-handler

PR feedback

291f6ff

jdpgrailsdev and others added 3 commits August 25, 2023 11:57

Store offset map in CDC state

31726f3

Formatting

ee781c1

Merge branch 'jonathan/cdc-metadata-injector' into jonathan/cdc-state…

28e4dc0

…-handler

disable checkpointing + saveStateAfterCompletionOfSnapshotOfNewStreams

971af01

subodh1810 approved these changes Aug 25, 2023

View reviewed changes

jdpgrailsdev and others added 2 commits August 28, 2023 09:45

Merge branch 'jonathan/cdc-metadata-injector' into jonathan/cdc-state…

9a13b5f

…-handler

Automated Commit - Formatting Changes

8743788

jdpgrailsdev merged commit da46171 into jonathan/cdc-metadata-injector Aug 28, 2023

jdpgrailsdev deleted the jonathan/cdc-state-handler branch August 28, 2023 15:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Source MongoDB Internal POC: CDC State Handling #29763

✨ Source MongoDB Internal POC: CDC State Handling #29763

jdpgrailsdev commented Aug 23, 2023

github-actions bot commented Aug 23, 2023 •

edited by jdpgrailsdev

Loading

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023 •

edited

Loading

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 24, 2023

github-actions bot commented Aug 24, 2023

github-actions bot commented Aug 24, 2023

jpefaur left a comment

subodh1810 left a comment

subodh1810 Aug 25, 2023

jdpgrailsdev Aug 25, 2023

jdpgrailsdev commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

subodh1810 left a comment

subodh1810 Aug 25, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

github-actions bot commented Aug 28, 2023

		import java.util.concurrent.atomic.AtomicLong;
		import org.bson.BsonTimestamp;

✨ Source MongoDB Internal POC: CDC State Handling #29763

✨ Source MongoDB Internal POC: CDC State Handling #29763

Conversation

jdpgrailsdev commented Aug 23, 2023

What

How

Recommended reading order

github-actions bot commented Aug 23, 2023 • edited by jdpgrailsdev Loading

Before Merging a Connector Pull Request

github-actions bot commented Aug 23, 2023

source-mssql test report (commit 3ab0fe24d6) - ❌

github-actions bot commented Aug 23, 2023 • edited Loading

Coverage report for source-postgres

github-actions bot commented Aug 23, 2023

source-mysql test report (commit 3ab0fe24d6) - ❌

github-actions bot commented Aug 23, 2023

source-mongodb-internal-poc test report (commit 3ab0fe24d6) - ❌

github-actions bot commented Aug 23, 2023

source-postgres test report (commit 3ab0fe24d6) - ❌

github-actions bot commented Aug 23, 2023

source-mongodb-internal-poc test report (commit 2f2fe533eb) - ❌

github-actions bot commented Aug 23, 2023

source-mysql test report (commit 2f2fe533eb) - ❌

github-actions bot commented Aug 23, 2023

source-postgres test report (commit 2f2fe533eb) - ❌

github-actions bot commented Aug 23, 2023

source-mssql test report (commit 2f2fe533eb) - ❌

github-actions bot commented Aug 23, 2023

source-mongodb-internal-poc test report (commit 9317ccfd6d) - ❌

github-actions bot commented Aug 23, 2023

source-mssql test report (commit 9317ccfd6d) - ❌

github-actions bot commented Aug 24, 2023

source-mongodb-internal-poc test report (commit 9517f34860) - ❌

github-actions bot commented Aug 24, 2023

source-mysql test report (commit cbf5f095e8) - ❌

github-actions bot commented Aug 24, 2023

source-postgres test report (commit cbf5f095e8) - ❌

jpefaur left a comment

Choose a reason for hiding this comment

subodh1810 left a comment

Choose a reason for hiding this comment

subodh1810 Aug 25, 2023

Choose a reason for hiding this comment

jdpgrailsdev Aug 25, 2023

Choose a reason for hiding this comment

jdpgrailsdev commented Aug 25, 2023

github-actions bot commented Aug 25, 2023

source-mongodb-internal-poc test report (commit af7feba553) - ❌

github-actions bot commented Aug 25, 2023

source-mongodb-internal-poc test report (commit 291f6ff50c) - ❌

github-actions bot commented Aug 25, 2023

source-mongodb-internal-poc test report (commit 291f6ff50c) - ❌

github-actions bot commented Aug 25, 2023

source-mysql test report (commit 28e4dc0380) - ❌

github-actions bot commented Aug 25, 2023

source-mongodb-internal-poc test report (commit 28e4dc0380) - ❌

github-actions bot commented Aug 25, 2023

source-postgres test report (commit 28e4dc0380) - ❌

github-actions bot commented Aug 25, 2023

source-mssql test report (commit 28e4dc0380) - ❌

subodh1810 left a comment

Choose a reason for hiding this comment

subodh1810 Aug 25, 2023

Choose a reason for hiding this comment

github-actions bot commented Aug 25, 2023

source-mongodb-internal-poc test report (commit 971af014e2) - ❌

github-actions bot commented Aug 25, 2023

source-mssql test report (commit 971af014e2) - ❌

github-actions bot commented Aug 25, 2023

source-mysql test report (commit 971af014e2) - ❌

github-actions bot commented Aug 25, 2023

source-postgres test report (commit 971af014e2) - ❌

github-actions bot commented Aug 28, 2023

source-mongodb-internal-poc test report (commit 9a13b5fae7) - ❌

github-actions bot commented Aug 23, 2023 •

edited by jdpgrailsdev

Loading

source-mssql test report (commit `3ab0fe24d6`) - ❌

github-actions bot commented Aug 23, 2023 •

edited

Loading

source-mysql test report (commit `3ab0fe24d6`) - ❌

source-mongodb-internal-poc test report (commit `3ab0fe24d6`) - ❌

source-postgres test report (commit `3ab0fe24d6`) - ❌

source-mongodb-internal-poc test report (commit `2f2fe533eb`) - ❌

source-mysql test report (commit `2f2fe533eb`) - ❌

source-postgres test report (commit `2f2fe533eb`) - ❌

source-mssql test report (commit `2f2fe533eb`) - ❌

source-mongodb-internal-poc test report (commit `9317ccfd6d`) - ❌

source-mssql test report (commit `9317ccfd6d`) - ❌

source-mongodb-internal-poc test report (commit `9517f34860`) - ❌

source-mysql test report (commit `cbf5f095e8`) - ❌

source-postgres test report (commit `cbf5f095e8`) - ❌

source-mongodb-internal-poc test report (commit `af7feba553`) - ❌

source-mongodb-internal-poc test report (commit `291f6ff50c`) - ❌

source-mongodb-internal-poc test report (commit `291f6ff50c`) - ❌

source-mysql test report (commit `28e4dc0380`) - ❌

source-mongodb-internal-poc test report (commit `28e4dc0380`) - ❌

source-postgres test report (commit `28e4dc0380`) - ❌

source-mssql test report (commit `28e4dc0380`) - ❌

source-mongodb-internal-poc test report (commit `971af014e2`) - ❌

source-mssql test report (commit `971af014e2`) - ❌

source-mysql test report (commit `971af014e2`) - ❌

source-postgres test report (commit `971af014e2`) - ❌

source-mongodb-internal-poc test report (commit `9a13b5fae7`) - ❌