-
Notifications
You must be signed in to change notification settings - Fork 121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add Connection interface #1374
Merged
stephaniewang526
merged 311 commits into
googleapis:main
from
stephaniewang526:query-interface
May 6, 2022
Merged
feat: add Connection interface #1374
stephaniewang526
merged 311 commits into
googleapis:main
from
stephaniewang526:query-interface
May 6, 2022
+6,381
−6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
google-cla
bot
added
the
cla: yes
This human has signed the Contributor License Agreement.
label
Jun 15, 2021
product-auto-label
bot
added
the
api: bigquery
Issues related to the googleapis/java-bigquery API.
label
Jun 15, 2021
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Neenu1995
added
the
kokoro:force-run
Add this label to force Kokoro to re-run the tests.
label
Jul 27, 2021
yoshi-kokoro
removed
the
kokoro:force-run
Add this label to force Kokoro to re-run the tests.
label
Jul 27, 2021
stephaniewang526
changed the title
feat: add QueryConnection interface
feat: add Connection interface
Aug 3, 2021
shollyman
reviewed
Aug 6, 2021
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryDryRunResult.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryResultSet.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuerySQLException.java
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/Connection.java
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/Connection.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ConnectionSettings.java
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/QueryResultRowFormat.java
Outdated
Show resolved
Hide resolved
stephaniewang526
added
the
kokoro:force-run
Add this label to force Kokoro to re-run the tests.
label
Aug 12, 2021
yoshi-kokoro
removed
the
kokoro:force-run
Add this label to force Kokoro to re-run the tests.
label
Aug 12, 2021
stephaniewang526
force-pushed
the
query-interface
branch
from
September 10, 2021 14:36
eb824b4
to
8935c10
Compare
stephaniewang526
force-pushed
the
query-interface
branch
from
September 17, 2021 18:08
54f0826
to
3595799
Compare
stephaniewang526
force-pushed
the
query-interface
branch
2 times, most recently
from
September 24, 2021 15:35
e2dc6b9
to
4584ef2
Compare
stephaniewang526
added
kokoro:force-run
Add this label to force Kokoro to re-run the tests.
and removed
kokoro:force-run
Add this label to force Kokoro to re-run the tests.
labels
Sep 24, 2021
yoshi-kokoro
removed
the
kokoro:force-run
Add this label to force Kokoro to re-run the tests.
label
Sep 27, 2021
google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITBigQueryTest.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITBigQueryTest.java
Show resolved
Hide resolved
google-cla
bot
added
cla: yes
This human has signed the Contributor License Agreement.
and removed
cla: yes
This human has signed the Contributor License Agreement.
cla: no
This human has *not* signed the Contributor License Agreement.
labels
May 5, 2022
stephaniewang526
added
the
owlbot:run
Add this label to trigger the Owlbot post processor.
label
May 5, 2022
gcf-owl-bot
bot
removed
the
owlbot:run
Add this label to trigger the Owlbot post processor.
label
May 5, 2022
google-cla
bot
added
cla: no
This human has *not* signed the Contributor License Agreement.
cla: yes
This human has signed the Contributor License Agreement.
and removed
cla: yes
This human has signed the Contributor License Agreement.
cla: no
This human has *not* signed the Contributor License Agreement.
labels
May 5, 2022
stephaniewang526
added
kokoro:force-run
Add this label to force Kokoro to re-run the tests.
kokoro:run
Add this label to force Kokoro to re-run the tests.
and removed
kokoro:force-run
Add this label to force Kokoro to re-run the tests.
labels
May 5, 2022
yoshi-kokoro
removed
the
kokoro:run
Add this label to force Kokoro to re-run the tests.
label
May 5, 2022
google-cla
bot
added
cla: no
This human has *not* signed the Contributor License Agreement.
and removed
cla: yes
This human has signed the Contributor License Agreement.
labels
May 6, 2022
stephaniewang526
added
the
owlbot:run
Add this label to trigger the Owlbot post processor.
label
May 6, 2022
gcf-owl-bot
bot
removed
the
owlbot:run
Add this label to trigger the Owlbot post processor.
label
May 6, 2022
google-cla
bot
added
cla: yes
This human has signed the Contributor License Agreement.
and removed
cla: no
This human has *not* signed the Contributor License Agreement.
labels
May 6, 2022
stephaniewang526
added a commit
to stephaniewang526/java-bigquery
that referenced
this pull request
May 6, 2022
…erystorage Java client Relates to googleapis#1374
gcf-merge-on-green bot
pushed a commit
that referenced
this pull request
May 10, 2022
🤖 I have created a release *beep* *boop* --- ## [2.11.0](v2.10.10...v2.11.0) (2022-05-10) ### Features * add Connection interface ([#1374](#1374)) ([3804275](3804275)) * next release from main branch is 2.10.9 ([#1996](#1996)) ([f716427](f716427)) ### Bug Fixes * add native image configuration for Arrow ([#2018](#2018)) ([06cbe69](06cbe69)) * fix for flaky connection close issue ([#2034](#2034)) ([db3daac](db3daac)) ### Documentation * **sample:** remove unused dependency and add setup instructions ([#2010](#2010)) ([e2e9113](e2e9113)) ### Dependencies * update dependency com.google.apis:google-api-services-bigquery to v2-rev20220422-1.32.1 ([#2017](#2017)) ([b9fa786](b9fa786)) * update dependency com.google.apis:google-api-services-bigquery to v2-rev20220429-1.32.1 ([#2020](#2020)) ([78789a5](78789a5)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Hi Read of 100_000 rows takes 23930 ms. Mono.fromCallable { bigQueryOptionsBuilder.build().service }
.flatMap { context ->
val connectionSettings = ConnectionSettings.newBuilder()
.setRequestTimeout(10L)
.setUseReadAPI(true)
.setMaxResults(1000)
.setNumBufferedRows(1000)
.setUseQueryCache(true)
.build();
val connection = context.createConnection(connectionSettings)
val bqResult = connection.executeSelect(sql)
val result = Flux.usingWhen(
Mono.just(bqResult.resultSet),
{ resultSet -> resultSet.toFlux(bqResult.schema) },
{ _ -> Mono.fromRunnable<Unit> { connection.close() } }
)
Mono.just(Data(result, bqResult.schema.toSchema()))
}
...
fun ResultSet.toFlux(schema:Schema): Flux<DataRecord> {
return Flux.generate<DataRecord> { sink ->
if (next()) {
sink.next(toDataRecord(schema))
} else {
sink.complete()
}
}
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
api: bigquery
Issues related to the googleapis/java-bigquery API.
cla: yes
This human has signed the Contributor License Agreement.
size: xl
Pull request size is extra large.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background: go/bq-sql-client-java
This PR provides a completely new Connection interface which defines separate APIs for different types of queries. This allows us to provide an industry standard way for database applications to build against the Java client library. We provide 3 new JDBC-esque API methods:
[In-scope for this PR] executeSelect - Only supports read only SELECT queries.
[In-scope for this PR] dryRun - returns som query processing statistics including schema and query parameters.
[Not in this PR] executeUpdate - Only supports DML and DDL.
[Not in this PR] execute - Any SQL - scripts, DML, DDL, SELECT etc statements.
We also integrate with the BigQueryStorage client library and use the high throughput Read API when applicable to parse query results using Arrow format. Arrow has shown better performance over Avro as the row serialization format in this experiment.