Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉 New Destination: Weaviate #20094
🎉 New Destination: Weaviate #20094
Changes from all commits
28c0395
77431fa
7b159bc
9e56857
228c7c5
875be1b
8c9ad5b
5be647f
1de0ca2
617ecf5
c8fd137
1845194
169025b
2f440ae
a6b35e3
3a898bf
88f6e61
c246173
d1f76d2
57e4432
ebabd2f
c9e5815
44d5ee0
54d08c8
5a935b8
4c1a46e
1472625
5c79c5a
2da6170
1915b6e
9173d23
28f3649
f190fc9
f0618e6
ac61e59
46d9525
c53714f
a5d8d23
b2d1fa3
69f9df0
85aeae8
8700d47
83b3a09
0d4e230
c71e3be
da2533d
f7bbdcf
53e84a9
0207834
6cd4503
dc1e7a9
08ab623
527560a
a8e3047
19541e9
d75438b
1befb54
6f8e6a1
2649a97
c421e34
d577f24
cce55e6
a10a1af
c0e8598
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think the replication should continue if the connector fails to replicate records. you should throw an exception and terminate the replication
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the case would be 1 of the objects in the batch had an error however other records might not have an error. So I was thinking we should continue but simply log an error for those. I was planning to add a retry mechanism in a follow up PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I'm aware that only a small amount of records can fail and the rest can be replicated successfully but so far the convention in the rest of the destination connectors was to terminate the replication and execute any cleanup logic such as removing temp tables and similar. tbh I'm on the fence for this, can you count a replication as successful overall if records were partially replicated 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've implemented partial batch retry and will now throw an exception if after 3 retries the there are still partial errors. Please review.