v3.0.0

This release contains a few improvements, bug fixes, and introduces GA support for Snowpipe Streaming to Snowflake managed Apache Iceberg™ tables

What's Changed

• SNOW-1708577 Parquet V2 support for new table format by @sfc-gh-alhuang in #851
• SNOW-1727532 Set number of values for repeated fields by @sfc-gh-alhuang in #861
• SNOW-1754295 - Prep for upcoming subscoped tokens PR by @sfc-gh-hmadan in #867
• SNOW-1748333 Fix Iceberg decimal type schema parser by @sfc-gh-alhuang in #864
• Fix empty map & list ingestion by @sfc-gh-alhuang in #873
• SNOW-1754295 - Start using subscoped tokens for iceberg ingestion by @sfc-gh-hmadan in #868
• SNOW-1768252 Ignore getStringForLogging in the requests by @sfc-gh-ggeng in #875
• SNOW-1760732 Enable Iceberg mode from properties by @sfc-gh-alhuang in #871
• SNOW-1764301 Iceberg parquet file configuration by @sfc-gh-alhuang in #874
• SNOW-1774276 Update fileId key in metadata for Iceberg mode by @sfc-gh-alhuang in #880
• Add enableIcebergStreaming to telemetry calls by @sfc-gh-hmadan in #878
• Re-merge "Add chunk offset to file id key to make each chunk have a unique key" from pull 825 by @sfc-gh-psaha in #865
• Enhance error messages in the SDK by @sfc-gh-hmadan in #883
• SNOW-1778664 Update snowflake-jdbc to 3.20.0 by @sfc-gh-lsembera in #887
• SNOW-1757554 Support quoted object fields name by @sfc-gh-alhuang in #869
• Fix quoted column name for Iceberg primitive type by @sfc-gh-alhuang in #891
• SNOW-1492090 Snowpipe streaming file master key id rotation by @sfc-gh-bmikaili in #786
• [SNOW-1711460] Upgrade to iceberg version 1.6.1 by @sfc-gh-xhuang in #894
• SNOW-1787322 Fix InsertError for structured data type by @sfc-gh-alhuang in #888
  ]
• Introduce Iceberg streaming merge gate by @sfc-gh-alhuang in #896
• NO_SNOW release new sdk version by @sfc-gh-xhuang in #893
• Explicitly specify avro dependency to override apache iceberg version by @sfc-gh-xhuang in #900

Full Changelog: v2.3.0...v3.0.0

v2.3.0

[Behavior Change Release]
This release changes how variant values represented as json strings are ingested into semi-structured columns (variant, object, array) if they contain numeric values. Starting with release 2.3.0, numeric values preserve their format, no conversion to and from scientific notation is happening anymore.

What's Changed

• Iceberg Ingestion in CloudStorage mode - Carve out IStorage and ExternalVolume from InternalStage by @sfc-gh-hmadan in #828
• SNOW-1659373 cleanup serializeFromParquetBuffers by @sfc-gh-gdoci in #829
• SNOW-1507007 Support schema for new table format by @sfc-gh-alhuang in #814
• Carve out MockSnowflakeServiceClient to allow ExternalVolume tests in next PR to use it properly by @sfc-gh-hmadan in #833
• SNOW-1673203 EP info generation for new table format by @sfc-gh-alhuang in #836
• SNOW-1674752 Fix vulnerable dependencies by @sfc-gh-alhuang in #838
• Get test coverage for telemetry reporting (was missing) + Fix SnowflakeServiceClient test code to allow per-API overrides by @sfc-gh-hmadan in #839
• SNOW-1675591 Fill in ExternalVolume and ExternalVolumeManager to do presigned url retrieval + blobname population by @sfc-gh-hmadan in #837
• Set createdon in BlobMetadata and is_iceberg in Drop Channel request by @sfc-gh-hmadan in #845
• SNOW-1666189 Structured data type support by @sfc-gh-alhuang in #841
• SNOW-1707031 hadoop upgrade for vulnerability fix by @sfc-gh-japatel in #849
• NO_SNOW Release 2.3.0 BCR by @sfc-gh-xhuang in #842
• NO-SNOW Remove unnecessary dependencies to reduce jar size by @sfc-gh-japatel in #852
• Fix concurrent null-ref in External Volume Blob path Generation + track URL timeout by @sfc-gh-hmadan in #854
• Fix testUtils to not repeatedly do keypair generation by @sfc-gh-hmadan in #855
• Add (non-public) method to expose iceberg schema for each column of a channel by @sfc-gh-hmadan in #856
• SNOW-1566045 Preserve numeric format for variants by @sfc-gh-lsembera in #817

Full Changelog: v2.2.2...v2.3.0

v2.2.2

What's Changed

• SNOW-1619347: upgrade bc-fips by @sfc-gh-tzhang in #810
• SNOW-1483230 Parameter support & disable blob encryption for new table format by @sfc-gh-alhuang in #801
• NO_SNOW Upgrade dependency versions by @sfc-gh-xhuang in #800
• SNOW-1497358 Support multiple stage for new table format by @sfc-gh-alhuang in #812
• SNOW-1654124: Write file name to metadata at the place when we create the file by @sfc-gh-tzhang in #824
• V2.2.2 Release by @sfc-gh-tzhang in #826

Full Changelog: v2.2.0...v2.2.2

v2.2.1

[Improvement] This release fixes a performance regression in upload

What's Changed

• SNOW-1619347: upgrade bc-fips by @sfc-gh-tzhang in #810
• SNOW-1483230 Parameter support & disable blob encryption for new table format by @sfc-gh-alhuang in #801
• NO_SNOW Upgrade dependency versions by @sfc-gh-xhuang in #800
• SNOW-1497358 Support multiple stage for new table format by @sfc-gh-alhuang in #812

Full Changelog: v2.2.0...v2.2.1

v2.2.0

[Bug Fix] Fix a critical bug that could potentially cause corruption with change_tracking enabled.
It is strongly recommend to upgrade to 2.2.0+ as the minimum version for all Snowpipe Streaming usage
[Improvement] Refactor some of the code logic to support different storage volume

What's Changed

• Code refactor for different storage volume support by @sfc-gh-alhuang in #792
• SNOW-1512047 Introduce independent per-table flushes when interleaving is disabled by @sfc-gh-alhuang in #788
• SNOW-1618257 Fix PRIMARY_FILE_ID_KEY by @sfc-gh-kkloudas in #807
• V2.2.0 Release by @sfc-gh-tzhang in #808

Full Changelog: v2.1.2...v2.2.0

v2.1.2

This release contains a few improvements and bug fixes for Snowpipe Streaming:

[Bug Fix] Flaky tests are improved
[Bug Fix] Fixed an issue with failover across deployments
[Improvement] InsertRows Performance improvements
[Improvement] Added or Improved various logs for better observability
[Improvement] Fine tune channel and chunk sizes

What's Changed

• Add error log for flush task scheduling by @sfc-gh-alhuang in #760
• SNOW-1435061 Upgrade JDBC driver for domains fix by @sfc-gh-lsembera in #766
• no-snow Parallelize e2e jar tests by @sfc-gh-lsembera in #767
• SNOW-1357377 Add request Id in all streaming ingest APIs by @sfc-gh-japatel in #759
• no-snow Fix flaky test by @sfc-gh-lsembera in #772
• SNOW-1457523: Fix CVE for snowflake-ingest-java io.airlift:aircompressor 0.21 by @sfc-gh-tzhang in #774
• Various performance improvements in the path by @sfc-gh-psaha in #782
• SNOW-1457523: Upgrade Parquet dependencies version to fix CVE by @sfc-gh-tzhang in #779
• SNOW-1465503 Check row count in Parquet footer before committing by @sfc-gh-lsembera in #784
• SNOW-1512935 Retry SocketTimeoutException by @sfc-gh-lsembera in #789
• NO-SNOW Fix flaky test using multiple threads by @sfc-gh-lsembera in #790
• SNOW-1373151: Proactively refresh token to avoid token expiration exception from JDBC by @sfc-gh-tzhang in #780
• Reject new stage metadata if the deployment id does not match what the client was created with by @sfc-gh-psaha in #794
• SNOW-1512935 Reduce client socket timeout from 5 minutes to 1 minute by @sfc-gh-lsembera in #795
• SNOW-1545879 Reduce the max channel/chunk sizes by @sfc-gh-lsembera in #796
• NO-SNOW Upgrade protobuf package by @sfc-gh-xhuang in #793
• V2.1.2 Release by @sfc-gh-asen in #797

v2.1.1

What's Changed

• NO-SNOW updated version by @sfc-gh-dshah in #686
• SNOW-1165817 SNOW-1165815 Update vulnerable dependencies by @sfc-gh-lsembera in #693
• SNOW-1230527 Disable linkage checker by @sfc-gh-lsembera in #717
• SNOW-1196967 SNOW-1196966 Update vulnerable dependencies by @sfc-gh-lsembera in #720
• SNOW-1255447: Fix leaked connection due to error response by @sfc-gh-tzhang in #719
• SNOW-1000828 Expose specific error messages as part of the channel invalidation exception by @sfc-gh-alhuang in #721
• SNOW-993600 External OAuth2.0 Support by @sfc-gh-alhuang in #718
• NO-SNOW Reduce OAuth test time by @sfc-gh-alhuang in #732
• SNOW-1258064 Remove/Relax the limitations in order to generate bigger files by @sfc-gh-alhuang in #730
• [Snyk] SNOW-1327598: Fix for 3 vulnerabilities by @sfc-gh-snowflakedb-snyk-sa in #734
• SNOW-1167007: Validate no null values for not null cols by @sfc-gh-wtrefon in #739
• [Snyk] Security upgrade org.bouncycastle:bcpkix-jdk18on from 1.78 to 1.78.1 by @sfc-gh-snowflakedb-snyk-sa in #752
• SNOW-1361785 Run e2e jar tests with multiple Java versions by @sfc-gh-lsembera in #756
• No-SNOW Update fasterxml by @sfc-gh-xhuang in #740
• SNOW-1262359 Upgrade commons-configuration2 by @sfc-gh-lsembera in #761
• V2.1.1 Release by @sfc-gh-tzhang in #758

v2.1.0

What's Changed

• NO-SNOW Correct invalid use of javadoc by @sfc-gh-xhuang in #668
• SNOW-983635 Allow ZSTD compression algorithm by @sfc-gh-lthiede in #654
• SNOW-1032874: Fix an Overflow Issue for Integer-store Timestamp by @sfc-gh-tzhang in #673
• Snowpipe Streaming: send column ordinal back to GS with blob EPs to register by @sfc-gh-azagrebin in #674
• SNOW-979849: Add start offset token at batch level in the Client SDK by @sfc-gh-tzhang in #676
• SNOW-967235 Sends start and end offsets from the Client SDK by @sfc-gh-tjones in #675
• NO-SNOW: Two minor changes that address PR comments and fix code format by @sfc-gh-tzhang in #679
• SNOW-987122 Upgrade JDBC to 3.14.5 and Catch new exception type for r… by @sfc-gh-japatel in #677
• Allow clients to drop channel on close or blindly by @sfc-gh-psaha in #657
• Add more detailed javadoc for the drop related api to warn about the … by @sfc-gh-psaha in #681
• SNOW-979849: Support user to pass a user defined offset token verification logic as part of channel creation by @sfc-gh-tzhang in #680
• V2.1.0 Release by @sfc-gh-tzhang in #682

New Contributors

• @sfc-gh-psaha made their first contribution in #657

Full Changelog: v2.0.5...v2.1.0

v2.0.5

This release contains a few improvements and bug fixes for Snowpipe Streaming:

[Improvement] Added an optional offset token parameter for openChannel.
[Improvement] Added support for specifying compression algorithm to be used for BDEC Parquet files.
[Improvement] Updated to support customized URL and added Snowflake account name in request header.
[Improvement] Implemented a change to send spansMixedTables flag in blob registration requests.
[Improvement] Deprecated BUFFER_FLUSH_INTERVAL_IN_MILLIS parameter, instead use the MAX_CLIENT_LAG parameter.
[Improvement] Implemented the refresh of downscoped GCS tokens.
[Bug Fix] Reverted one change that updated public API for internal use case.
[Bug Fix] Fixed the end-to-end JAR test so it can run on all cloud platforms.

What's Changed

• Add an optional offset token parameter for openChannel by @sfc-gh-asen in #645
• support specifying compression algorithm to be used for BDEC Parquet files by @sfc-gh-gdoci in #579
• Disable Snowpipe flaky tests by @sfc-gh-tzhang in #647
• changed snapshot version from 2.0.4 to 2.0.5-SNAPSHOT by @sfc-gh-dshah in #650
• benchmark setup by @sfc-gh-lthiede in #655
• Update to support customized url and add customer name in request header by @sfc-gh-dwang in #656
• send spansMixedTables flag in blob registration requests by @sfc-gh-gdoci in #651
• Fix name of database the dataset is created in by @sfc-gh-lthiede in #658
• NO-SNOW: Deprecate BUFFER_FLUSH_INTERVAL_IN_MILLIS parameter by @sfc-gh-tzhang in #659
• NO-SNOW Run e2e jar tests on all clouds by @sfc-gh-lsembera in #666
• NO-SNOW: Revert one change that updates public API for internal use case by @sfc-gh-tzhang in #662
• GCS token refresh by @sfc-gh-lsembera in #665
• V2.0.5 release by @sfc-gh-tzhang in #664

Full Changelog: v2.0.4...v2.0.5

v2.0.4

This release contains a few improvements and bug fixes for Snowpipe Streaming:

• [Improvement] Support a new ON_ERROR option SKIP_BATCH, which will skip the entire batch if there is any issue and return all errors as part of the response
• [Improvement] Add row index information to all exceptions
• [Improvement] Upgrade snappy-java dependency
• [Improvement] Add a new interface to return the table schema information for a channel
• [Improvement] Add a new configuration option MAX_CLIENT_LAG which can contain the flush frequency, by default we flush every second
• [Bug Fix] Fix an issue when using snowflake-jdbc-fips
• [Bug Fix] Fix a rare ConcurrentModificationException issue
• [Bug Fix] Fix two issues in insertRows API that might cause wrong results in a very rare case
• [Bug Fix] Limit the max allowed number of chunks in blob to avoid the case when the request is too large

What's Changed

• Remove Public Preview note by @sfc-gh-lsembera in #582
• @no-snow refactor a few variables into ClientBufferParameters by @sfc-gh-gdoci in #581
• SNOW-914666 Adds MAX_CLIENT_LAG configuration option by @sfc-gh-tjones in #586
• SNOW-919423 Surfaces Table Schema for a Channel by @sfc-gh-tjones in #589
• NO-SNOW Remove wrong documentation by @sfc-gh-lthiede in #591
• updated SDK version to 2.0.4-SNAPSHOT by @sfc-gh-dshah in #588
• SNOW-924864 Upgrade snappy-java by @sfc-gh-lsembera in #592
• SNOW-918949: Add row index information to all exceptions by @sfc-gh-tzhang in #590
• SNOW-928802 Use concurrent hashmap to prevent ConcurrentModificationE… by @sfc-gh-japatel in #594
• SNOW-926149 Fix issues while using snowflake-jdbc-fips by @sfc-gh-lsembera in #596
• Update readme for JDBC version issues #599 by @sfc-gh-xhuang in #600
• SNOW-906224: Add new ON_ERROR=SKIP_BATCH option by @sfc-gh-tzhang in #597
• NO-SNOW: Fix two issues in insertRows API by @sfc-gh-tzhang in #602
• SNOW-902709 Limit the max allowed number of chunks in blob by @sfc-gh-lsembera in #580
• PRODSEC-3611 fix GHA parsing by @sfc-gh-jfan in #603
• no-snow Sort pom.xml in format.sh by @sfc-gh-lsembera in #604
• SNOW-936900 Use upgraded version of snappy-java by @sfc-gh-lsembera in #605
• SNOW-945927 SNOW-945928 Fix Snyk vulnerabilities by @sfc-gh-lsembera in #641
• NO-SNOW: fix row index issue for ON_ERROR=SKIP_BATCH by @sfc-gh-tzhang in #606
• V2.0.4 release by @sfc-gh-tzhang in #643

New Contributors

• @sfc-gh-lthiede made their first contribution in #591

Full Changelog: v2.0.3...v2.0.4