Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery: Raise helpful error when loading table from dataframe with STRUCT columns #9053

Merged
merged 5 commits into from
Aug 21, 2019

Conversation

plamut
Copy link
Contributor

@plamut plamut commented Aug 18, 2019

Closes #9024.

This PR detects if loading table data from a dataframe with STRUCT columns, and raises a helpful error before doing any more work.

It also issues a pending deprecation warning if schema is not provided as discussed separately.

Edit: Now seeing the specs for no-schema deprecation in #9042, this PR can be viewed as a step towards it (the additional logic for when to issue a warning will be added separately). If the warning should be removed altogether in this PR, please let me know.

How to test

  • Create a table matching the schema from the issue description.
  • Run the example from the issue description.

Expected result (after the fix):
Instead of an uninformative FileNotFoundError, a ValueError with a clear message is raised.

@plamut plamut added the api: bigquery Issues related to the BigQuery API. label Aug 18, 2019
@plamut plamut requested review from tswast and a team August 18, 2019 22:54
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Aug 18, 2019
Copy link
Contributor

@tswast tswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Struct column error LGTM once we update the message.

bigquery/google/cloud/bigquery/client.py Outdated Show resolved Hide resolved
bigquery/google/cloud/bigquery/client.py Outdated Show resolved Hide resolved
plamut added 2 commits August 21, 2019 09:18
The warning will be added once the support for partial schemas and
automatic schema detection is implemented.
@plamut plamut requested a review from tswast August 21, 2019 07:25
@plamut plamut merged commit 405e009 into googleapis:master Aug 21, 2019
@plamut plamut deleted the iss-9024 branch August 21, 2019 16:24
HemangChothani pushed a commit to HemangChothani/google-cloud-python that referenced this pull request Aug 29, 2019
…STRUCT columns (googleapis#9053)

* Issue warning if no schema when loading from DF

* Raise error if serializing DF with struct fields

* Rewrite test assertion to make coverage happy

* Make the unsupported type message more general

* Remove warning on missing schema

The warning will be added once the support for partial schemas and
automatic schema detection is implemented.
emar-kar pushed a commit to MaxxleLLC/google-cloud-python that referenced this pull request Sep 18, 2019
…STRUCT columns (googleapis#9053)

* Issue warning if no schema when loading from DF

* Raise error if serializing DF with struct fields

* Rewrite test assertion to make coverage happy

* Make the unsupported type message more general

* Remove warning on missing schema

The warning will be added once the support for partial schemas and
automatic schema detection is implemented.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BigQuery: 'load_table_from_dataframe' raises OSError with STRUCT / RECORD columns.
3 participants