test: improve `to_gbq` logic unit test coverage #449

tswast · 2021-12-20T23:25:43Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Towards #392 🦕

~~Based on #443~~

…o-query

…ecessary

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…ssue392-unit-test-100

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…-bigquery-pandas into issue392-unit-test-100

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…-bigquery-pandas into issue392-unit-test-100

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast · 2021-12-23T23:12:47Z

pandas_gbq/gbq.py

@@ -1057,7 +1063,7 @@ def to_gbq(
                DeprecationWarning,
                stacklevel=2,
            )
-        elif api_method == "load_csv":
+        else:


There's already a check above for known api_methods, so this was an impossible branch.

tswast · 2021-12-23T23:13:00Z

pandas_gbq/gbq.py

@@ -1122,12 +1128,14 @@ def to_gbq(
            )
        elif if_exists == "replace":
            connector.delete_and_recreate_table(dataset_id, table_id, table_schema)
-        elif if_exists == "append":
+        else:


There's already a check above for known if_exists, so this was an impossible branch.

tswast · 2021-12-23T23:13:35Z

pandas_gbq/gbq.py

            if not pandas_gbq.schema.schema_is_subset(original_schema, table_schema):
                raise InvalidSchema(
                    "Please verify that the structure and "
                    "data types in the DataFrame match the "
-                    "schema of the destination table."
+                    "schema of the destination table.",
+                    table_schema,


Towards #349

tswast · 2021-12-23T23:16:57Z

.coveragerc

@@ -22,7 +22,7 @@ omit =
  google/cloud/__init__.py

 [report]
-fail_under = 89
+fail_under = 95


Still not 100, but it's solid progress. I don't want this PR to get too long. Mostly I started working on unit tests because they're just mindless enough that I can handle it while I'm sick this week.

It's fine to merge a meaningful piece of work into main, it doesn't have to be all-or-nothing. This PR by itself is already a significant improvement.

(I will review this tomorrow, did not manage to go through everything today)

tswast · 2021-12-23T23:18:08Z

tests/unit/conftest.py

-    def get_table(table_ref_or_id, **kwargs):
-        return google.cloud.bigquery.Table(table_ref_or_id)
-
-    mock_client.get_table.side_effect = get_table


This was messing with the to_gbq tests, so moved to test_gbq.py (and potentially to a test_read_gbq.py in future.

plamut

Looking good, not much to add really, one or two remarks aside.

One of the system tests might need an adjustment, though, it does not look like flakiness.

pandas_gbq/gbq.py

tests/unit/test_load.py

tswast · 2021-12-28T16:44:11Z

I've removed the failing system test. The behavior of _Table.delete() did in fact change, but it's not a part of the public interface and the new behavior is verified by unit tests.

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast added 18 commits December 6, 2021 16:22

feat: accepts a table ID, which downloads the table without a query

9a9d3fd

add todo for next steps

6adf233

Merge remote-tracking branch 'upstream/main' into issue266-read_gbq-n…

73a791a

…o-query

add unit test for table ID read_gbq

9b1eb0d

add helper for is_query

ec9ddaf

implement read_gbq with table id

9cc7c74

fix remaining tests, don't localalize out-of-bounds timestamp columns

dd51ad8

Update pandas_gbq/gbq.py

e1ad679

fix 3.7 unit tests

d29bc2a

correct coverage

cb8f24f

skip coverage for optional test skip

56b73b2

fix docs build

8a61e97

improve test coverage for error case

3f7900b

as of google-cloud-bigquery 1.11.0, get_table before list_rows is unn…

3c53f1f

…ecessary

tests with whitespace

5ce125f

type annotations

ea660f4

test: improve unit test coverage

3d93c78

boost coverage

93e872e

product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-pandas API. label Dec 20, 2021

gcf-owl-bot bot and others added 11 commits December 20, 2021 23:27

🦉 Updates from OwlBot

e0ae455

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

boost coverage

28b72f0

Merge remote-tracking branch 'upstream/issue392-unit-test-100' into i…

ed61f6d

…ssue392-unit-test-100

🦉 Updates from OwlBot

1d6831f

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

finish coverage for load.py

fcf8276

Merge branch 'issue392-unit-test-100' of github.com:googleapis/python…

dfb107e

…-bigquery-pandas into issue392-unit-test-100

another test

76b38a3

refactor gbq tests

2fd1e32

less intense refactoring

d97102e

Merge remote-tracking branch 'upstream/main' into issue392-unit-test-100

c1f8055

more refactor cleanup

c48f997

tswast and others added 7 commits December 23, 2021 15:56

add to_gbq tests

83c4513

boost coverage

4f12c78

🦉 Updates from OwlBot

4fedaaf

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

cover new properties

2bfd5a1

Merge branch 'issue392-unit-test-100' of github.com:googleapis/python…

24574a8

…-bigquery-pandas into issue392-unit-test-100

unknown if_exists

3cb788e

add session for no deps

95f0478

tswast changed the title ~~test: improve unit test coverage~~ test: improve to_gbq logic unit test coverage Dec 23, 2021

tswast marked this pull request as ready for review December 23, 2021 23:11

tswast requested a review from a team December 23, 2021 23:11

tswast requested a review from a team as a code owner December 23, 2021 23:11

tswast requested review from shollyman and plamut December 23, 2021 23:11

🦉 Updates from OwlBot

0f9baa8

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast commented Dec 23, 2021

View reviewed changes

tswast mentioned this pull request Dec 23, 2021

Improve InvalidSchema exception by adding specific fields that do not match #349

Open

tswast commented Dec 23, 2021

View reviewed changes

plamut reviewed Dec 28, 2021

View reviewed changes

pandas_gbq/gbq.py Show resolved Hide resolved

tests/unit/test_load.py Show resolved Hide resolved

tswast added 2 commits December 28, 2021 10:39

remove system test for private delete method

7ac9b6a

check number of columns

d562ee9

tswast requested a review from plamut December 28, 2021 16:43

tswast and others added 2 commits December 28, 2021 11:32

coverage dropped due to removed code

5599915

🦉 Updates from OwlBot

5223fa4

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

plamut approved these changes Dec 28, 2021

View reviewed changes

tswast merged commit 3ae5d4c into main Dec 28, 2021

tswast deleted the issue392-unit-test-100 branch December 28, 2021 20:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: improve `to_gbq` logic unit test coverage #449

test: improve `to_gbq` logic unit test coverage #449

tswast commented Dec 20, 2021 •

edited

Loading

tswast Dec 23, 2021

tswast Dec 23, 2021

tswast Dec 23, 2021

tswast Dec 23, 2021

plamut Dec 27, 2021 •

edited

Loading

tswast Dec 23, 2021

plamut left a comment •

edited

Loading

tswast commented Dec 28, 2021

test: improve to_gbq logic unit test coverage #449

test: improve to_gbq logic unit test coverage #449

Conversation

tswast commented Dec 20, 2021 • edited Loading

tswast Dec 23, 2021

Choose a reason for hiding this comment

tswast Dec 23, 2021

Choose a reason for hiding this comment

tswast Dec 23, 2021

Choose a reason for hiding this comment

tswast Dec 23, 2021

Choose a reason for hiding this comment

plamut Dec 27, 2021 • edited Loading

Choose a reason for hiding this comment

tswast Dec 23, 2021

Choose a reason for hiding this comment

plamut left a comment • edited Loading

Choose a reason for hiding this comment

tswast commented Dec 28, 2021

test: improve `to_gbq` logic unit test coverage #449

test: improve `to_gbq` logic unit test coverage #449

tswast commented Dec 20, 2021 •

edited

Loading

plamut Dec 27, 2021 •

edited

Loading

plamut left a comment •

edited

Loading