Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deps!: BigQuery Storage and pyarrow are required dependencies #776

Merged
merged 27 commits into from
Jul 27, 2021
Merged
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
ba49697
process: make BQ Storage and pyarrow required
plamut Jul 14, 2021
2c79152
Make pyarrow required in _pandas_helpers.py
plamut Jul 14, 2021
137ae92
Make pyarrow required in client.py
plamut Jul 15, 2021
42c9606
Make pyarrow required in table.py
plamut Jul 16, 2021
5ae720d
Make pyarrow required in job/query.py
plamut Jul 16, 2021
ede0313
Make pyarrow required in DBAPI tests
plamut Jul 16, 2021
ce7f93d
Make pyarrow required in snippets tests
plamut Jul 16, 2021
3f7c456
Make BQ storage required in client.py
plamut Jul 16, 2021
ed412c0
Make BQ storage required in table.py
plamut Jul 16, 2021
ef905c2
Make BQ storage required in DB API tests
plamut Jul 16, 2021
9c106f3
Make BQ storage required in magics.py
plamut Jul 16, 2021
5438239
Make BQ storage required in test__helpers.py
plamut Jul 16, 2021
cfcc2d3
Make BQ storage required in test__pandas_helpers.py
plamut Jul 16, 2021
86c4533
Make BQ storage required in test_query_pandas.py
plamut Jul 16, 2021
560ccd6
Make method signatures compatible again
plamut Jul 16, 2021
e658d5a
Remove checks for minimum BQ Storage version
plamut Jul 19, 2021
777b850
Merge branch 'master' into iss-757
plamut Jul 20, 2021
70572d8
Remove LegacyBigQueryStorageError
plamut Jul 20, 2021
be5cf32
Bump minimum pyarrow version to 3.0.0
plamut Jul 21, 2021
7960729
Remove unneeded pytest.importorskip for BQ Storage
plamut Jul 21, 2021
b5475d5
Merge branch 'master' into iss-757
plamut Jul 22, 2021
c0b810b
Remove pyarrow version checks in pandas helpers tests
plamut Jul 22, 2021
8d127cf
Conditionally skip pandas tests where needed
plamut Jul 22, 2021
e47f1bd
Remove unneeded conditional pyarrow version paths
plamut Jul 22, 2021
427665e
Cover schema autodetect failed code path in test
plamut Jul 22, 2021
820b092
Merge branch 'master' into iss-757
tswast Jul 27, 2021
c4315ce
fix bad merge
tswast Jul 27, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' into iss-757
  • Loading branch information
plamut committed Jul 22, 2021
commit b5475d5cf8c3732f963e0314e572315990c276b2
14 changes: 0 additions & 14 deletions google/cloud/bigquery/client.py
Original file line number Diff line number Diff line change
@@ -27,7 +27,6 @@
import json
import math
import os
import packaging.version
import tempfile
from typing import Any, BinaryIO, Dict, Iterable, Optional, Sequence, Tuple, Union
import uuid
@@ -109,9 +108,6 @@
# https://github.com/googleapis/python-bigquery/issues/438
_MIN_GET_QUERY_RESULTS_TIMEOUT = 120

# https://github.com/googleapis/python-bigquery/issues/781#issuecomment-883497414
_PYARROW_BAD_VERSIONS = frozenset([packaging.version.Version("2.0.0")])


class Project(object):
"""Wrapper for resource describing a BigQuery project.
@@ -2576,16 +2572,6 @@ def load_table_from_dataframe(
try:

if job_config.source_format == job.SourceFormat.PARQUET:
if _PYARROW_VERSION in _PYARROW_BAD_VERSIONS:
msg = (
"Loading dataframe data in PARQUET format with pyarrow "
f"{_PYARROW_VERSION} can result in data corruption. It is "
"therefore *strongly* advised to use a different pyarrow "
"version or a different source format. "
"See: https://github.com/googleapis/python-bigquery/issues/781"
)
warnings.warn(msg, category=RuntimeWarning)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warning was removed because the just-fixed bug will not come into play once we require pyarrow > 3.0.0.


if job_config.schema:
if parquet_compression == "snappy": # adjust the default value
parquet_compression = parquet_compression.upper()
1 change: 0 additions & 1 deletion tests/unit/test_client.py
Original file line number Diff line number Diff line change
@@ -27,7 +27,6 @@
import warnings

import mock
import packaging
import requests
import pytest
import pytz
You are viewing a condensed version of this merge commit. You can view the full changes here.