Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vector DB CDK: Delete cdc records #32496

Merged
merged 5 commits into from
Nov 16, 2023
Merged

Vector DB CDK: Delete cdc records #32496

merged 5 commits into from
Nov 16, 2023

Conversation

flash1293
Copy link
Contributor

Fixes #29827

This PR looks for the _ab_cdc_deleted_at field in records which signifies that this record got deleted in the source and does not produce any chunks out of it but deletes it in the destination

@flash1293 flash1293 requested a review from a team as a code owner November 14, 2023 11:39
Copy link

vercel bot commented Nov 14, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Nov 16, 2023 3:16pm

@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label Nov 14, 2023
Copy link
Collaborator

@aaronsteers aaronsteers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks, @flash1293 !

if current_stream.primary_key and current_stream.destination_sync_mode == DestinationSyncMode.append_dedup:
metadata[METADATA_RECORD_ID_FIELD] = f"{stream_identifier}_{self._extract_primary_key(record, current_stream)}"
return metadata
if not current_stream.primary_key or not current_stream.destination_sync_mode == DestinationSyncMode.append_dedup:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit... Generally != or "is not" would be preferable for the "not equals" comparison.

@flash1293 flash1293 enabled auto-merge (squash) November 16, 2023 15:14
@flash1293 flash1293 merged commit aa111d2 into master Nov 16, 2023
12 checks passed
@flash1293 flash1293 deleted the flash1293/vector-db-cdc branch November 16, 2023 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CDK Connector Development Kit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Vector db based CDC: Support deleting documents of deleted records for CDC sources
3 participants