deps!: BigQuery Storage and pyarrow are required dependencies #776

plamut · 2021-07-16T14:17:49Z

Closes #757.

This might already be a final version, but I still need to check a few additional things:

~~Remove logic for "ensure client"? Or maybe not, since we still allow tabledata.list?~~
We still need that.
~~Remove min version check for BQ Storage, since now it is guaranteed >=2.0.0 will be installed (no more pip quirks with extras).~~
Done.
~~Remove checks for GAPIC? It's now a required dependency.~~
Not applicable, the minimum GAPIC version was never checked at runtime.

PR checklist::

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

The annotations caused a mismatch

Since this is now a required dependency, there should not be any more pip quirks that used to allow installing BQ Storage as an extra, but without always respecting its minimum version pin.

plamut · 2021-07-19T20:31:49Z

@tswast It might be easier to review each commit separately, I tried to make them independent from each other. Also, double check if the assumption stated in the last commit's message really holds - if not, we can revert that bit.

tseaver · 2021-07-19T20:32:38Z

setup.py

    "google-cloud-core >= 1.4.1, < 3.0dev",
    "google-resumable-media >= 0.6.0, < 3.0dev",
    "packaging >= 14.3",
    "protobuf >= 3.12.0",
+    "pyarrow >= 1.0.0, < 5.0dev",


Making these previously-optional dependencies now required feels like it might be a semver: major change.

IIRC @tswast said in a chat that it shouldn't be a problem for most users, and those who cannot install these dependencies for any reason can still pin to a lower (non-major version). But reading this as I write, we might actually want to re-consider this.

There has been a debate on this on the issue itself. Also copying Tim's input:

The gist is that the problematic dependencies are already required transitively anyway
With pyarrow being the exception
It's probably worth the bump for pyarrow if we're being conservative about it. Esp since we do know of environments where adding it will break

Let's go with the major version bump. I'd rather people adopt this change a little more slowly than break someone who has a working 2.x branch without pyarrow

Roger.

Is there perhaps anything else that we want to clean up and would be a breaking change? (possibly in a separate PR)
A major version bump is a good opportunity for that.

#319 is still on my mind. Perhaps easier now that proto-plus added more case options?

Another breaking change we might want to do: https://issuetracker.google.com/144712110 -- Better type conversions in the pandas connector.

~~And yet another one from today - maybe bumping the minimum pyarrow version, as the older ones suffer from a bug (load data from dataframe can upload incorrect data, yikes!)~~

Update: Done.

That is a nasty bug. Thankfully pyarrow didn't follow NEP-29 this time and kept support for Python 3.6 in the 4.x packages. I'd be okay with bumping the minimum for that.

And yet another - if this gets delayed significantly (read: months), we might want to consider dropping Python 3.6 support as well. But Python 3.6 end of life will only happen in December this year.

google/cloud/bigquery/exceptions.py

setup.py

plamut · 2021-07-21T16:50:26Z

As a follow-up to #781 I also bumped the minimum pyarrow version.

tseaver · 2021-07-21T17:07:54Z

Failing systest is another instance of #733.

plamut · 2021-07-22T14:48:39Z

google/cloud/bigquery/client.py

-                        "version or a different source format. "
-                        "See: https://github.com/googleapis/python-bigquery/issues/781"
-                    )
-                    warnings.warn(msg, category=RuntimeWarning)


This warning was removed because the just-fixed bug will not come into play once we require pyarrow > 3.0.0.

tswast

This is awesome!

FYI: I pointed this PR at the v3 branch. Should be safe to merge.

deps!: BigQuery Storage and pyarrow are required dependencies (#776) fix!: use nullable `Int64` and `boolean` dtypes in `to_dataframe` (#786) feat!: destination tables are no-longer removed by `create_job` (#891) feat!: In `to_dataframe`, use `dbdate` and `dbtime` dtypes from db-dtypes package for BigQuery DATE and TIME columns (#972) fix!: automatically convert out-of-bounds dates in `to_dataframe`, remove `date_as_object` argument (#972) feat!: mark the package as type-checked (#1058) feat!: default to DATETIME type when loading timezone-naive datetimes from Pandas (#1061) feat: add `api_method` parameter to `Client.query` to select `INSERT` or `QUERY` API (#967) fix: improve type annotations for mypy validation (#1081) feat: use `StandardSqlField` class for `Model.feature_columns` and `Model.label_columns` (#1117) docs: Add migration guide from version 2.x to 3.x (#1027) Release-As: 3.0.0

deps!: BigQuery Storage and pyarrow are required dependencies (googleapis#776) fix!: use nullable `Int64` and `boolean` dtypes in `to_dataframe` (googleapis#786) feat!: destination tables are no-longer removed by `create_job` (googleapis#891) feat!: In `to_dataframe`, use `dbdate` and `dbtime` dtypes from db-dtypes package for BigQuery DATE and TIME columns (googleapis#972) fix!: automatically convert out-of-bounds dates in `to_dataframe`, remove `date_as_object` argument (googleapis#972) feat!: mark the package as type-checked (googleapis#1058) feat!: default to DATETIME type when loading timezone-naive datetimes from Pandas (googleapis#1061) feat: add `api_method` parameter to `Client.query` to select `INSERT` or `QUERY` API (googleapis#967) fix: improve type annotations for mypy validation (googleapis#1081) feat: use `StandardSqlField` class for `Model.feature_columns` and `Model.label_columns` (googleapis#1117) docs: Add migration guide from version 2.x to 3.x (googleapis#1027) Release-As: 3.0.0

product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Jul 16, 2021

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Jul 16, 2021

tswast changed the title ~~process: make BigQUery Storage and pyarrow required dependencies~~ deps: BigQuery Storage and pyarrow are required dependencies Jul 16, 2021

plamut added 16 commits July 19, 2021 11:16

process: make BQ Storage and pyarrow required

ba49697

Make pyarrow required in _pandas_helpers.py

2c79152

Make pyarrow required in client.py

137ae92

Make pyarrow required in table.py

42c9606

Make pyarrow required in job/query.py

5ae720d

Make pyarrow required in DBAPI tests

ede0313

Make pyarrow required in snippets tests

ce7f93d

Make BQ storage required in client.py

3f7c456

Make BQ storage required in table.py

ed412c0

Make BQ storage required in DB API tests

ef905c2

Make BQ storage required in magics.py

9c106f3

Make BQ storage required in test__helpers.py

5438239

Make BQ storage required in test__pandas_helpers.py

cfcc2d3

Make BQ storage required in test_query_pandas.py

86c4533

Make method signatures compatible again

560ccd6

The annotations caused a mismatch

Remove checks for minimum BQ Storage version

e658d5a

Since this is now a required dependency, there should not be any more pip quirks that used to allow installing BQ Storage as an extra, but without always respecting its minimum version pin.

plamut force-pushed the iss-757 branch from d7f0531 to e658d5a Compare July 19, 2021 20:28

plamut marked this pull request as ready for review July 19, 2021 20:29

plamut requested a review from a team July 19, 2021 20:29

plamut requested a review from a team as a code owner July 19, 2021 20:29

plamut requested review from loferris and tswast and removed request for a team July 19, 2021 20:29

tseaver reviewed Jul 19, 2021

View reviewed changes

plamut commented Jul 19, 2021

View reviewed changes

google/cloud/bigquery/exceptions.py Outdated Show resolved Hide resolved

plamut commented Jul 20, 2021

View reviewed changes

setup.py Show resolved Hide resolved

plamut added 2 commits July 21, 2021 18:45

Bump minimum pyarrow version to 3.0.0

be5cf32

Remove unneeded pytest.importorskip for BQ Storage

7960729

plamut added 5 commits July 22, 2021 13:53

Merge branch 'master' into iss-757

b5475d5

Remove pyarrow version checks in pandas helpers tests

c0b810b

Conditionally skip pandas tests where needed

8d127cf

Remove unneeded conditional pyarrow version paths

e47f1bd

Cover schema autodetect failed code path in test

427665e

plamut commented Jul 22, 2021

View reviewed changes

tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 22, 2021

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 22, 2021

plamut requested a review from tswast July 23, 2021 06:36

tswast added 2 commits July 27, 2021 12:10

Merge branch 'master' into iss-757

820b092

fix bad merge

c4315ce

tswast changed the base branch from master to v3.x.x July 27, 2021 17:21

tswast changed the base branch from v3.x.x to v3 July 27, 2021 17:28

tswast approved these changes Jul 27, 2021

View reviewed changes

tswast merged commit 9cd7554 into googleapis:v3 Jul 27, 2021

plamut deleted the iss-757 branch July 28, 2021 07:17

tswast mentioned this pull request Oct 6, 2021

pull google-cloud-bigquery-storage and pyarrow into required dependencies #757

Closed

tswast mentioned this pull request Mar 29, 2022

fix!: remove out-of-date BigQuery ML protocol buffers #1178

Merged

4 tasks

release-please bot mentioned this pull request Mar 29, 2022

chore(main): release 3.0.0 #1179

Merged

tswast mentioned this pull request Apr 13, 2022

New dependency on pyarrow introduces heavyweight numpy sub-dependency #1196

Closed

steffnay mentioned this pull request Jul 15, 2022

deps: make pyarrow and BQ Storage optional dependencies #1282

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deps!: BigQuery Storage and pyarrow are required dependencies #776

deps!: BigQuery Storage and pyarrow are required dependencies #776

plamut commented Jul 16, 2021 •

edited

Loading

plamut commented Jul 19, 2021

tseaver Jul 19, 2021

plamut Jul 19, 2021

plamut Jul 19, 2021

tswast Jul 19, 2021

plamut Jul 20, 2021

tswast Jul 20, 2021

tswast Jul 20, 2021

plamut Jul 20, 2021 •

edited

Loading

tswast Jul 20, 2021

plamut Jul 21, 2021

plamut commented Jul 21, 2021

tseaver commented Jul 21, 2021

plamut Jul 22, 2021

tswast left a comment

deps!: BigQuery Storage and pyarrow are required dependencies #776

deps!: BigQuery Storage and pyarrow are required dependencies #776

Conversation

plamut commented Jul 16, 2021 • edited Loading

plamut commented Jul 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

plamut Jul 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

plamut commented Jul 21, 2021

tseaver commented Jul 21, 2021

Choose a reason for hiding this comment

tswast left a comment

Choose a reason for hiding this comment

plamut commented Jul 16, 2021 •

edited

Loading

plamut Jul 20, 2021 •

edited

Loading