-
Notifications
You must be signed in to change notification settings - Fork 513
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider using google-cloud-bigquery library instead of google-api-services-bigquery #1555
Comments
Update: the client library bug affecting extract jobs has been fixed! googleapis/google-cloud-java#3924 |
@clairemcginty is this still worth looking? |
Talked IRL, closing. |
I would like us to reconsider re-opening this. I think there's still some subtle bugs in our current internal BigQuery client. Some of these bugs are related to not fallbacking to env setting properties. |
@nevillelyh @clairemcginty what was the reason to not go forward with this? |
@regadas If I remember right, it was due to the complexity of integrating with Beam's BigQuery sources/sinks -- Beam returned types from This was awhile ago though, so maybe worth a second look? |
@clairemcginty interesting! I think it's worth looking into it again since we are already using the Let's see if the other types are good to go as well. I'll book some time to look into this. Thanks |
Google documentation recommends using the client library
google-cloud-bigquery
rather than the API librarygoogle-api-services-bigquery
.Pros
google-cloud-bigquery
uses typed protobuf API request/response params rather than plain strings/ints, implicitly handles transport layer configurations, and potentially improves performance by making direct RPC calls rather than JSON over HTTP. We've seen intermittent failures with the scio-bigquery IT suite due to network timeouts, which might be solved by migrating.google-api-services-bigquery
library is in maintenance mode and aside from critical bug fixes, won't have any new features added.Cons
TableSchema
,TableReference
, andTableRow
- map toSchema
,TableId
, andFieldValueList
in the client library. So, if we end up migrating, we'd have to decide whether to change the externally facing Scio API or handle those conversions ourselves in private methods. I have a WIP branch for this migration I'll link to as soon as it's cleaned up.The text was updated successfully, but these errors were encountered: