You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If your output format for a SQLToGCSOperator is json, then any "dict" type object returned from a database, for example a Postgres JSON column, is not dumped to a string and is kept as a nested JSON object.
Add option to dump dict objects to string in JSON exporter.
Use case/motivation
Currently JSON type columns are hard to ingest into BQ since a JSON field in a source database does not enforce a schema, and we can't reliably generate a RECORD schema for the column.
Also, somewhat unrelated, the schema generated if a column is of type "JSON" is for a column of type "STRING". If you try to load the data into BigQuery using the generated schema it will fail if you don't dump the dictionaries to string first.
Description
If your output format for a SQLToGCSOperator is
json
, then any "dict" type object returned from a database, for example a Postgres JSON column, is not dumped to a string and is kept as a nested JSON object.Add option to dump
dict
objects to string in JSON exporter.Use case/motivation
Currently JSON type columns are hard to ingest into BQ since a JSON field in a source database does not enforce a schema, and we can't reliably generate a
RECORD
schema for the column.Related issues
No response
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: