-
Notifications
You must be signed in to change notification settings - Fork 196
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement at-least-once option that utilizes default stream #1007
Implement at-least-once option that utilizes default stream #1007
Conversation
/gcbrun |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please add an integration test for testing the writeAtLeastOnce
configuration?
a5b0ad5
to
1c48241
Compare
Added an integration test to exercise the following scenarios with writeAtLeastOnce turned on: (a) append to a new file |
1c48241
to
015c4ed
Compare
015c4ed
to
95b81a6
Compare
...onnector-common/src/main/java/com/google/cloud/bigquery/connector/common/BigQueryClient.java
Outdated
Show resolved
Hide resolved
@@ -208,6 +208,23 @@ public boolean deleteTable(TableId tableId) { | |||
return bigQuery.delete(tableId); | |||
} | |||
|
|||
private Job buildQueryJob( | |||
TableId temporaryTableId, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be sourceTableId
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
...onnector-common/src/main/java/com/google/cloud/bigquery/connector/common/BigQueryClient.java
Show resolved
Hide resolved
9a12708
to
ddcb019
Compare
/gcbrun |
ddcb019
to
469d279
Compare
/gcbrun |
1 similar comment
/gcbrun |
000429b
to
b7cb303
Compare
/gcbrun |
} | ||
|
||
protected void writeToBigQuery(Dataset<Row> df, SaveMode mode, String format) { | ||
protected void writeToBigQuery(Dataset<Row> df, SaveMode mode, String writeAtLeastOnce) { | ||
writeToBigQuery(df, mode, "avro", writeAtLeastOnce); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please maintain the format and avoid overriding it - to maintain backward compatibility this should be
protected void writeToBigQuery(Dataset<Row> df, SaveMode mode, String format) {
writeToBigQuery(df, mode, format, "False");
}
@@ -162,17 +162,23 @@ private StandardTableDefinition testPartitionedTableDefinition() { | |||
} | |||
|
|||
protected void writeToBigQuery(Dataset<Row> df, SaveMode mode) { | |||
writeToBigQuery(df, mode, "avro"); | |||
writeToBigQuery(df, mode, "False"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
replace with writeToBigQuery(df, mode, "avro", "False");
to support streaming scenarios in which small batches of data are written.
b7cb303
to
af17ebe
Compare
/gcbrun |
to support streaming scenarios in which small batches of data are written.