v0.16.1-beta.1

What's Changed

New Features 🎉

• feat: make scalar index training configurable by @dsgibbons in #2686
• feat: migrate Bitmap & LabelList index file to v2 format by @BubbleCal in #2677
• feat: make default batch size configurable via environment variable by @westonpace in #2693

Full Changelog: v0.16.0...v0.16.1-beta.1

v0.16.0

What's Changed

Breaking Changes 🛠

• feat!: simplify take row api by @eddyxu in #2664
• feat!: distinguishable scalar index types by @eddyxu in #2671

New Features 🎉

• feat: standalone vector transform stage by @westonpace in #2566
• feat: coalesce scheduling of reads to speed up random access by @raunaks13 in #2636
• feat: add version tags by @dsgibbons in #2482
• feat: object store registry for custom object store providers by @maxburke in #2513
• feat: merge_insert update subcolumns by @wjones127 in #2639
• feat: implement disk-based inverted index by @BubbleCal in #2643
• feat: allow round-tripping of dictionary data through the v2 format by @westonpace in #2656
• feat: support transforming selected fragments in vector transform stage for ivf_pq index by @raunaks13 in #2657
• feat: expand tags api by @dsgibbons in #2679
• feat: add standalone shuffle for transformed ivf-pq vectors file by @raunaks13 in #2670
• feat: support bitpacking for signed types by @albertlockett in #2662
• feat: support loading huggingface image dataset and convert image to PIL by @eddyxu in #2684
• feat: return BM25 scores for FTS by @BubbleCal in #2687
• feat: add data file format / version information to manifest by @westonpace in #2673
• feat: add support for the null data type to v2 by @westonpace in #2685
• feat: add backpressure to v2 I/O scheduler by @westonpace in #2683

Bug Fixes 🐛

• fix: correctly encode a list type when all items are empty by @westonpace in #2653
• fix: improve error message when can't train PQ on too small dataset by @albertlockett in #2644
• fix: slight cleanups to path handling so that the indices builder tool properly supports Windows by @westonpace in #2689

Documentation 📚

• docs: schema evolution by @wjones127 in #1911
• docs: reorg the scalar index python docstring to make the index type clear by @eddyxu in #2678

Performance Improvements 🚀

• perf: add v2 fragment file metadata to the FileMetadataCache by @jiachengdb in #2647
• perf: add random take benchmark by @chebbyChefNEQ in #2654
• perf: benchmark lance vs parquet read time, write time, and compressed size by @raunaks13 in #2383

Other Changes

• refactor: new buffer abstractions in decoders by @westonpace in #2648

New Contributors

• @dsgibbons made their first contribution in #2482
• @maxburke made their first contribution in #2513

Full Changelog: v0.15.0...v0.16.0

v0.15.1-beta.1

What's Changed

New Features 🎉

• feat: standalone vector transform stage by @westonpace in #2566

Bug Fixes 🐛

• fix: correctly encode a list type when all items are empty by @westonpace in #2653

Performance Improvements 🚀

• perf: add v2 fragment file metadata to the FileMetadataCache by @jiachengdb in #2647
• perf: add random take benchmark by @chebbyChefNEQ in #2654

Other Changes

• refactor: new buffer abstractions in decoders by @westonpace in #2648

Full Changelog: v0.15.0...v0.15.1-beta.1

v0.15.0: Experimental FTS, packed struct encoding

What's Changed

Breaking Changes 🛠

• feat: upgrade DataFusion, Arrow, PyO3, ObjectStore by @wjones127 in #2594

New Features 🎉

New feature breakdown

• feat: add fast search to to_table method by @chebbyChefNEQ in #2591
• feat: loosen pyarrow pin by @wjones127 in #2597
• feat: add FSST string compression by @broccoliSpicy in #2470
• feat: integrate inverted index into lance index APIs by @BubbleCal in #2577
  • You can now create full text search indices and pass text search queries into the full_text_query argument
• feat: add projection to take and take_rows by @westonpace in #2623
• feat: add storage_options to _BaseLanceDatasink, LanceDatasink, LanceCommitter by @BitPhinix in #2619
• feat: add bitpack encoding for LanceV2 by @albertlockett in #2333
• feat: add a packed struct encoding to lance by @raunaks13 in #2593
• feat: support deletion with stable row id and scalar indices by @wjones127 in #2609

Bug Fixes 🐛

• fix: avoid making empty range requests as these will err on S3 by @westonpace in #2592
• fix: upgrade AWS SDK by @wjones127 in #2616
• fix: specify behavior version when creating dynamodb config by @westonpace in #2624
• fix: pin numpy version to avoid numpy >= 2.0 until pyarrow supports it by @westonpace in #2637

Performance Improvements 🚀

• perf: improve inverted index performance by @BubbleCal in #2574
• perf: improve v2 scan performance by @westonpace in #2604
• perf: allow fragment scan for nearest query if there is a prefilter by @jiachengdb in #2631
• perf: speed up fsst decompression by @broccoliSpicy in #2626
• perf: skip documents with WAND by @BubbleCal in #2632

Other Changes

• refactor: move planner.rs, logical_expr.rs, and sql.rs from lance to lance_datafusion by @westonpace in #2622
• refactor: remove custom ProjectionExec by @wjones127 in #2608

New Contributors

• @BitPhinix made their first contribution in #2619

Full Changelog: v0.14.1...v0.15.0

v0.15.0-beta.2

What's Changed

New Features 🎉

• feat: add FSST string compression by @broccoliSpicy in #2470
• feat: integrate inverted index into lance index APIs by @BubbleCal in #2577
• feat: add projection to take and take_rows by @westonpace in #2623

Bug Fixes 🐛

• fix: upgrade AWS SDK by @wjones127 in #2616
• fix: specify behavior version when creating dynamodb config by @westonpace in #2624

Other Changes

• refactor: move planner.rs, logical_expr.rs, and sql.rs from lance to lance_datafusion by @westonpace in #2622

Full Changelog: v0.15.0-beta.1...v0.15.0-beta.2

v0.15.0-beta.1

What's Changed

Breaking Changes 🛠

• feat: upgrade DataFusion, Arrow, PyO3, ObjectStore by @wjones127 in #2594

New Features 🎉

• feat: loosen pyarrow pin by @wjones127 in #2597

Performance Improvements 🚀

• perf: improve inverted index performance by @BubbleCal in #2574
• perf: improve v2 scan performance by @westonpace in #2604

Full Changelog: v0.14.2-beta.1...v0.15.0-beta.1

v0.14.2-beta.1

What's Changed

New Features 🎉

• feat: add fast search to to_table method by @chebbyChefNEQ in #2591

Bug Fixes 🐛

• fix: avoid making empty range requests as these will err on S3 by @westonpace in #2592

Full Changelog: v0.14.1...v0.14.2-beta.1

v0.14.1

What's Changed

New Features 🎉

• feat: add inverted index by @BubbleCal in #2526
• feat(python): expose fast search flag in python by @eddyxu in #2573
• feat: use hyperloglog for cardinality estimation for dictionary encoding by @niyue in #2555
• feat: add scalar bitmap index to lance by @raunaks13 in #2560
• feat: add a standalone method for calculating the PQ model by @westonpace in #2562
• feat: integrate bitmap index into top-level lance APIs by @westonpace in #2575
• feat: support to set ef for search from python by @BubbleCal in #2569
• feat: dataset update accepts binary value by @eddyxu in #2579
• feat: do flat search if too many rows are filtered out by @BubbleCal in #2583
• feat: add label list index by @westonpace in #2581
• feat: allow passing in precomputed centroids to lance.util.KMeans by @jiachengdb in #2586
• feat: add standalone partition assignment operation by @westonpace in #2556

Bug Fixes 🐛

• fix: add support for list fields with names other than 'item' by @westonpace in #2580
• fix: propagate storage_options to v2 writer by @jiachengdb in #2578
• fix: be compatible with v1 index format by @BubbleCal in #2582
• fix: search panic with prefilter and HNSW by @BubbleCal in #2584

Performance Improvements 🚀

• perf: change scalar index to return RowIdTreeMap instead of u64 array by @westonpace in #2587

Full Changelog: v0.14.0...v0.14.1

v0.14.1-beta.2

What's Changed

New Features 🎉

• feat: support to set ef for search from python by @BubbleCal in #2569

Full Changelog: v0.14.1-beta.1...v0.14.1-beta.2

v0.14.1-beta.1

What's Changed

New Features 🎉

• feat: add inverted index by @BubbleCal in #2526
• feat(python): expose fast search flag in python by @eddyxu in #2573
• feat: use hyperloglog for cardinality estimation for dictionary encoding by @niyue in #2555
• feat: add scalar bitmap index to lance by @raunaks13 in #2560
• feat: add a standalone method for calculating the PQ model by @westonpace in #2562
• feat: integrate bitmap index into top-level lance APIs by @westonpace in #2575

Full Changelog: v0.14.0...v0.14.1-beta.1