Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add config option for snapshot time #1221

Merged

Conversation

tom-s-powell
Copy link
Contributor

No description provided.

Copy link
Member

@davidrabinowitz davidrabinowitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! Please see the comment below.

@@ -218,6 +218,7 @@ public static WriteMethod from(@Nullable String writeMethod) {
com.google.common.base.Optional.absent();
private boolean enableReadSessionCaching = true;
private long readSessionCacheDurationMins = 5L;
private OptionalLong snapshotTimeMillis = OptionalLong.empty();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SparkBigQueryConfig is a Serializable class, and therefore we need to keep all of it's members Serializable as well. This is why all Optionals are the guava version and not Java's. Please replace this OptionalLong with com.google.common.base.Optional<Long>? The conversion to OptionalLong should be done in getSnapshotTimeMillis()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have switched to have similar behaviour to partitionExpirationMs

@davidrabinowitz
Copy link
Member

/gcbrun

@davidrabinowitz davidrabinowitz merged commit 5691e24 into GoogleCloudDataproc:master May 6, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants