Releases: GoogleCloudDataproc/spark-bigquery-connector
Releases · GoogleCloudDataproc/spark-bigquery-connector
0.21.0
Bug Fixes
- Issue #354: users can query a view with different columns in select() and filter()
- Issue #367: Struct column order is fixed
- Issue #383: Fixed table metadata update when writing to a partitioned table
- Issue #390: Allowing additional white-space types in the query
- Issue #393: replacing avro.shaded dependency with guava
- PR #360: Removed redundant
UNNEST
when compilingIN
condition
Dependency Updates
- BigQuery API has been upgraded to version 1.131.1
- BigQuery Storage API has been upgraded to version 1.22.0
- Guava has been upgraded to version 30.1.1-jre
- gRPC has been upgraded to version 1.37.1
- Netty has been upgraded to version 4.1.65.Final
0.20.0
New Features
- PR #375: Added support for pseudo column support - time partitioned table now support the _PARTITIONTIME and _PARTITIONDATE fields
Bug Fixes
- Issue #190: Writing data to BigQuery properly populate the field description
- Issue #265: Fixed nested conjunctions/disjunctions when using the AVRO read format
- Issue #326: Fixing netty_tcnative_windows.dll shading
Dependency Updates
- Arrow has een upgraded to version 4.0.0
0.19.1
Bug Fixes
- PR #324 - Restoring version 0.18.1 dependencies due to networking issues
Dependency Updates
- BigQuery API has been upgraded to version 1.123.2
- BigQuery Storage API has been upgraded to version 1.6.0
- Guava has been upgraded to version 30.0-jre
- Netty has been upgraded to version 4.1.51.Final
0.19.0
New Features
- Issue #247: Allowing to load results of any arbitrary SELECT query from BigQuery.
- Issue #310: Allowing to configure the expiration time of materialized data.
- PR #283: Implemented Datasource v2 write support.
- Improved Spark 3 compatibility.
Dependency Updates
- BigQuery API has been upgraded to version 1.127.4
- BigQuery Storage API has been upgraded to version 1.10.0
- Guava has been upgraded to version 30.1-jre
- Netty has been upgraded to version 4.1.52.Final
0.18.1
New Features
- PR #276: Added the option to enable
useAvroLogicalTypes
option When writing data to BigQuery.
Bug Fixes
0.18.0
New Features
- Issue #226: Adding support for HOUR, MONTH, DAY Time Partitions
- Issue #260: Increasing connection timeout to the BigQuery service, and configuring the request retry settings.
Bug Fixes
- Issue #263: Fixed
select *
error when ColumnarBatch is used (DataSource v2) - Issue #266: Fixed the external configuration not working regression bug (Introduced in version 0.17.2)
- PR #262: Filters on BigQuery DATE and TIMESTAMP now use the right type.
Dependency Updates
- BigQuery API has been upgraded to version 1.123.2
- BigQuery Storage API has been upgraded to version 1.6.0
- Guava has been upgraded to version 30.0-jre
- Netty has been upgraded to version 4.1.51.Final
- netty-tcnative has been upgraded to version 4.1.34.Final
0.17.3
0.17.2
0.17.1
New Features
- PR #229: Adding support for Spark ML Vector and Matrix data types
Bug Fixes
- Issue #216: removed redundant ALPN dependency
- Issue #219: Fixed the LessThanOrEqual filter SQL compilation in the DataSource v2 implementation
- Issue #221: Fixed ProtobufUtilsTest.java with newer BigQuery dependencies
Dependency Updates
- BigQuery API has been upgraded to version 1.116.8
- BigQuery Storage API has been upgraded to version 1.3.1
0.17.0
New Features
- Structured streaming write is now supported (PR #201, thanks @varundhussa)
- Users now has the option to keep the data on GCS after writing to BigQuery (PR #202, thanks @leoneuwald)
- Enabling to overwrite data of a single date partition (PR #211)
- Supporting
MATERIALIZED_VIEW
as table type (PR #192) - Supporting columnar batch reads from Spark in the DataSource V2 implementation. (PR #198) It is not ready for production use.
Bug Fixes
- Conditions on StructType fields are now handled by Spark and not the connector, Fixing Issue #197
Dependency Updates
- BigQuery API has been upgraded to version 1.116.3
- BigQuery Storage API has been upgraded to version 1.0.0
- Netty has been upgraded to version 4.1.48.Final (Fixing issue #200)