Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Integer based partition Support #1065

Closed
slice-amandata opened this issue Sep 18, 2023 · 3 comments
Closed

Add Integer based partition Support #1065

slice-amandata opened this issue Sep 18, 2023 · 3 comments

Comments

@slice-amandata
Copy link

No description provided.

@vishalkarve15
Copy link
Contributor

@amandaolens we do already have Integer range based partition support.
We are bound by the partitioning offered by BigQuery, and it is limited to one of two types:

Integer range partitioning, as described also in issue #867
Date/Time partitioning (daily, hourly, monthly, or yearly)

@slice-amandata
Copy link
Author

@vishalkarve15 can you share the reference for integer based partition

@vishalkarve15
Copy link
Contributor

vishalkarve15 commented Sep 21, 2023

It's not released yet, you can refer to this for now: https://github.com/GoogleCloudDataproc/spark-bigquery-connector/blob/master/README-template.md (See partitionField option)
You can use the nightly builds to try it out for now (based on your spark version):

gs://spark-lib-nightly-snapshots/spark-2.4-bigquery-0.0.20230919.jar
gs://spark-lib-nightly-snapshots/spark-3.1-bigquery-0.0.20230919.jar
gs://spark-lib-nightly-snapshots/spark-3.2-bigquery-0.0.20230919.jar
gs://spark-lib-nightly-snapshots/spark-3.3-bigquery-0.0.20230919.jar
gs://spark-lib-nightly-snapshots/spark-3.4-bigquery-0.0.20230919.jar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants