Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery: Allow partial schema in load_table_from_dataframe #8140

Closed
tswast opened this issue May 24, 2019 · 1 comment · Fixed by #9064
Closed

BigQuery: Allow partial schema in load_table_from_dataframe #8140

tswast opened this issue May 24, 2019 · 1 comment · Fixed by #9064
Assignees
Labels
api: bigquery Issues related to the BigQuery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@tswast
Copy link
Contributor

tswast commented May 24, 2019

As discussed in #8105 (comment) and implemented in pandas-gbq at googleapis/python-bigquery-pandas#218, often times the default schema detection works well enough for pandas -> BQ, and only one or two of many columns need to be overridden.

Example:

df = pandas.DataFrame({... lots of columns ..., "needs_manual_schema": [...]})
job_config = bigquery.LoadJobConfig(schema=[
    bigquery.SchemaField("needs_manual_schema", "BOOLEAN"),
])
client.load_table_from_dataframe(
    df, "my_dataset.my_table", job_config=job_config,
)
# All columns from df are uploaded, but only df["needs_manual_schema"] is
# coerced into a specific type (BOOLEAN).
@tswast
Copy link
Contributor Author

tswast commented Aug 16, 2019

Now that I think about this, #9044 may be considered a sub-issue of this. Partial schemas make the most sense if we can determine the rest of the schema automatically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
1 participant