Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery: pyarrow warning about schema argument #9485

Closed
tswast opened this issue Oct 16, 2019 · 2 comments · Fixed by #9504
Closed

BigQuery: pyarrow warning about schema argument #9485

tswast opened this issue Oct 16, 2019 · 2 comments · Fixed by #9504
Assignees
Labels
api: bigquery Issues related to the BigQuery API. type: cleanup An internal cleanup or hygiene concern.

Comments

@tswast
Copy link
Contributor

tswast commented Oct 16, 2019

samples/tests/test_query_to_arrow.py::test_query_to_arrow
/tmpfs/src/github/google-cloud-python/bigquery/google/cloud/bigquery/_pandas_helpers.py:383: FutureWarning: Schema passed to names= option, please pass schema= explicitly. Will raise exception in future
return pyarrow.RecordBatch.from_arrays(arrays, column_names)

@tswast tswast added api: bigquery Issues related to the BigQuery API. type: cleanup An internal cleanup or hygiene concern. labels Oct 16, 2019
@les69
Copy link

les69 commented Oct 18, 2019

Hello everyone! I'm experiencing the same issue non-deterministically and it seems to be linked to a reduction in the downloading performance (from ~200k rows per second to ~20k).
Is there any workaround for the moment?

Thank you

@plamut
Copy link
Contributor

plamut commented Oct 20, 2019

@les69 In which pyarrow version does that happen? Does it also happen in an earlier version, e.g. 0.14.1 (i.e. the performance regression, not the warning)?

The warning itself seems pretty benign, it's just the way parameters are passed to one of the pyarrow's methods, and pyarrow resolves that on its own (but will eventually stop doing that).

It might be, however, that performance is different when using a pyarrow schema as opposed to just the column names (source), and the distinction between these two cases became more visible with the warning introduced in pyarrow 1.15.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. type: cleanup An internal cleanup or hygiene concern.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants