v0.18.3-beta.1

What's Changed

New Features 🎉

• feat: add a blob encoding for large binary values by @westonpace in #2868
• feat: cuda acceleration for PQ builds/assignments by @jacketsj in #2946
• feat(rust): add checkout_latest to Dataset by @wjones127 in #2991
• feat: support storage options for spark read and write by @SaintBacchus in #2990
• feat: add some basic primitive for working with repdef data by @westonpace in #2998
• feat: one-pass IVF_PQ accelerated builds by @jacketsj in #3001
• feat: add bit-width, cardinality and data-size to datablock statistics by @broccoliSpicy in #2986
• feat: add table config by @dsgibbons in #2820
• feat: expose number of rows covered per delta by @chebbyChefNEQ in #2979
• feat: add the basic encode path for 2.1 by @westonpace in #3002

Bug Fixes 🐛

• fix: filter the vec index in function Index::scalar_index_info by @SaintBacchus in #3000

Full Changelog: v0.18.2...v0.18.3-beta.1

v0.18.2

What's Changed

New Features 🎉

• feat: add a python API for reading arrays of binary/large_binary as file objects by @westonpace in #2944

Bug Fixes 🐛

• fix: don't use direct dependencies for cuvs as this prevents pypi from publishing by @westonpace in #2974

Performance Improvements 🚀

• perf: add python benchmark test for write/read integers by @broccoliSpicy in #2940

Full Changelog: v0.18.1...v0.18.2

v0.18.1

What's Changed

New Features 🎉

• feat: make Debug for useful Session by @wjones127 in #2903
• feat: log the execution plan in a more compact fashion by @westonpace in #2906
• feat(java): add dataset open storage options by @LuQQiu in #2871
• feat: reconnect block compression by @westonpace in #2878
• feat: handle nulls when creating indices with cuda by @westonpace in #2910
• feat: cuvs acceleration for gpu k-means by @jacketsj in #2816
• feat: allow users to specify late/early materialization, adjust default threshold by @westonpace in #2916
• feat: add an optional process-wide limit for IOPS by @westonpace in #2928
• feat: allow scalar indices to be disabled for a query by @westonpace in #2937
• feat: reconnect pushdown to v2 by @westonpace in #2913
• feat: support fastlanes bitpacking by @broccoliSpicy in #2886
• feat: nested list access by @wjones127 in #2966
• feat: add backwards compatibility with 0.13.0 v2 datasets by @westonpace in #2755

Bug Fixes 🐛

• fix: enlarge cargo release timeout and add workflow dispatch by @LuQQiu in #2901
• fix: fix cargo release tag checkout by @LuQQiu in #2902
• fix: restore ability to write fixed size list of nullable by @westonpace in #2907
• fix: use retry loop when loading deletion files by @westonpace in #2899
• fix: fix dot distance for cuda by @westonpace in #2914
• fix: use existing storage version on overwrite, if none specified by @westonpace in #2924
• fix: support negative values in SQL array literals by @wjones127 in #2925
• fix: v2 deadlock when there are multiple data files by @westonpace in #2923
• fix: refine_factor should be used in search_in_partition for IVF V2 Indices by @erikml-db in #2933
• fix: list_indices never updated after first call by @BubbleCal in #2936
• fix: postfilter should be applied in full if knn/fts even if scalar index applies by @westonpace in #2931
• fix: fix ray sink error when there are no data to write by @SaintBacchus in #2919
• fix: remove dup cache consts and respect index size param by @BubbleCal in #2948
• fix: don't always include first doc by @wjones127 in #2957
• fix: better error for concurrent create by @wjones127 in #2962
• fix: inline datafusion_common by @LuQQiu in #2963
• fix: include the list item field schema by @westonpace in #2950

New Contributors

• @erikml-db made their first contribution in #2933
• @SaintBacchus made their first contribution in #2919
• @alexwilcoxson-rel made their first contribution in #2917

Full Changelog: v0.18.0...v0.18.1

v0.18.1-beta.4

What's Changed

New Features 🎉

• feat: allow users to specify late/early materialization, adjust default threshold by @westonpace in #2916
• feat: add an optional process-wide limit for IOPS by @westonpace in #2928

Bug Fixes 🐛

• fix: support negative values in SQL array literals by @wjones127 in #2925
• fix: v2 deadlock when there are multiple data files by @westonpace in #2923

Full Changelog: v0.18.1-beta.3...v0.18.1-beta.4

v0.18.1-beta.3

What's Changed

New Features 🎉

• feat: cuvs acceleration for gpu k-means by @jacketsj in #2816

Bug Fixes 🐛

• fix: fix dot distance for cuda by @westonpace in #2914
• fix: use existing storage version on overwrite, if none specified by @westonpace in #2924

Full Changelog: v0.18.1-beta.2...v0.18.1-beta.3

v0.18.1-beta.2

What's Changed

New Features 🎉

• feat: make Debug for useful Session by @wjones127 in #2903
• feat: log the execution plan in a more compact fashion by @westonpace in #2906
• feat(java): add dataset open storage options by @LuQQiu in #2871
• feat: reconnect block compression by @westonpace in #2878
• feat: handle nulls when creating indices with cuda by @westonpace in #2910

Bug Fixes 🐛

• fix: restore ability to write fixed size list of nullable by @westonpace in #2907
• fix: use retry loop when loading deletion files by @westonpace in #2899

Full Changelog: v0.18.1-beta.1...v0.18.1-beta.2

v0.18.1-beta.1

What's Changed

Bug Fixes 🐛

• fix: enlarge cargo release timeout and add workflow dispatch by @LuQQiu in #2901
• fix: fix cargo release tag checkout by @LuQQiu in #2902

Full Changelog: v0.18.0...v0.18.1-beta.1

v0.18.0

What's Changed

Breaking Changes 🛠

• feat!: change the default data storage version to "stable" (e.g. v2.0) by @westonpace in #2829
• feat!: update_table returns stats by @QianZhu in #2867
• Revert "chore!: remove manylinux 2014 build (#2891)" by @eddyxu in #2892
• feat!: add batch_size option to merge_columns by @westonpace in #2896

New Features 🎉

• feat: propagate storage_options to LanceFileWriter and LanceFileReader by @ankitvij-db in #2840
• feat(java): support Lance Spark Batch Write by @LuQQiu in #2500
• feat: add a separate retry loop on object contents download by @westonpace in #2874
• feat: support decimal 128 and 256 json types by @LuQQiu in #2880
• feat: add CreateIndex commit type to python API by @jiachengdb in #2883

Bug Fixes 🐛

• fix: ensure the v2 file writer does not mix up the order of a column by @westonpace in #2836
• fix: calculate decode priority correctly by @westonpace in #2841
• fix: vector column mismatch should not be a IO Error by @LuQQiu in #2845
• fix: phrase query may return incorrect results if there's unknown word by @BubbleCal in #2850
• fix: check label list list column type earlier by @LuQQiu in #2846
• fix: can't filter on string column which has FTS index by @BubbleCal in #2875
• fix: limit/offset pushdown was not working correctly in the presence of deletions by @westonpace in #2895
• fix: fix release versions by @LuQQiu in #2897
• fix: revert Bump version f3eef61 by @LuQQiu in #2900

Performance Improvements 🚀

• perf: parallelize remapping to improve FTS compaction by @BubbleCal in #2834
• perf: concurrent loading FTS index files by @BubbleCal in #2787

Other Changes

• refactor: convert encoders to use data blocks by @westonpace in #2817

Full Changelog: v0.17.0...v0.18.0

v0.18.0-beta.2

Full Changelog: v0.18.0-beta.1...v0.18.0-beta.2

v0.18.0-beta.1

What's Changed

Breaking Changes 🛠

• feat!: change the default data storage version to "stable" (e.g. v2.0) by @westonpace in #2829

New Features 🎉

• feat: propagate storage_options to LanceFileWriter and LanceFileReader by @ankitvij-db in #2840
• feat(java): support Lance Spark Batch Write by @LuQQiu in #2500
• feat: add a separate retry loop on object contents download by @westonpace in #2874

Bug Fixes 🐛

• fix: vector column mismatch should not be a IO Error by @LuQQiu in #2845
• fix: phrase query may return incorrect results if there's unknown word by @BubbleCal in #2850
• fix: check label list list column type earlier by @LuQQiu in #2846
• fix: can't filter on string column which has FTS index by @BubbleCal in #2875

Other Changes

• refactor: convert encoders to use data blocks by @westonpace in #2817

Full Changelog: v0.17.1-beta.2...v0.18.0-beta.1