Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to our fork of emr-dynamodb-connector #183

Merged
merged 4 commits into from
Aug 11, 2024

Conversation

julienrf
Copy link
Collaborator

@julienrf julienrf commented Jul 24, 2024

Fixes #167

@julienrf
Copy link
Collaborator Author

julienrf commented Jul 24, 2024

Related: #117 (comment)

@julienrf
Copy link
Collaborator Author

I tested this PR locally and manually checked that #167 is still fixed by checking the absence of logs containing “BatchWriteItem”.
We also have our own test suite DynamoDBInputFormatTest checking that #130 is still fixed.

@julienrf julienrf marked this pull request as ready for review July 29, 2024 09:52
@julienrf
Copy link
Collaborator Author

The test failure is caused by a difference in the way we compute segments, see scylladb/emr-dynamodb-connector#8

@julienrf
Copy link
Collaborator Author

This PR is ready for review (and merge!).

In case we merge it, the next release version should be 1.0.0. If needed, we could keep a branch 0.9.x alive to publish patch release from there.

- Remove unnecessary logger configuration
- Use the `DynamoDBInputFormat` class from the fork instead of maintaining our copy
@julienrf julienrf force-pushed the use-dynamodb-connector-fork branch from bb26c4d to 3b2f53c Compare August 11, 2024 09:05
@tarzanek tarzanek merged commit e570a02 into scylladb:master Aug 11, 2024
1 check passed
@julienrf julienrf deleted the use-dynamodb-connector-fork branch August 14, 2024 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

$SPARK_HOME/work can fill up on worker nodes during DynamoDB migration
2 participants