Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for structs and modes (repeated and required) for BigQuery #313

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

faahren
Copy link

@faahren faahren commented Aug 21, 2024

Description & motivation

The BigQuery schema declaration was not supporting nested fields (STRUCT, or RECORD) as well as the REPEATED option that gives possibilty to store an array. Those are interesting when linking external tables more complex than CSV (Json, Parquet, etc)

Also, all fields were nullable by default and we could not set them to REQUIRED.

I updated the macro to allow full support for those 3 features
I have not updated the readme but added a sample for bigquery in the dedicated file.
I can't add a test as I do not have access to the bucket containing external samples (To add a nested JSON for example)

Checklist

  • [ X] I have verified that these changes work locally
  • [X ] I have updated the README.md (if applicable ) SAMPLES ADDED TO THE SAMPLE FILE
  • I have added an integration test for my fix/feature (if applicable)

@faahren
Copy link
Author

faahren commented Sep 3, 2024

@jeremyyeo Any idea when someone can look into this ?

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant