feat!: add batch_size option to merge_columns

[westonpace]

There's a few other places where updater is used (e.g. merge_insert / add_columns) and we may want to review those paths as well.

This is technically a breaking change on the rust API and so I will mark it as such.

BREAKING CHANGE: Fragment::updater now accepts a new batch_size argument

[westonpace]

Closes #2893

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 85.29412% with 5 lines in your changes missing coverage. Please review.

Project coverage is 77.92%. Comparing base (78d90a9) to head (743d810).

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/fragment.rs	50.00%	0 Missing and 2 partials ⚠️
rust/lance/src/dataset/updater.rs	87.50%	1 Missing and 1 partial ⚠️
rust/lance/src/dataset/scanner.rs	91.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2896      +/-   ##
==========================================
- Coverage   77.93%   77.92%   -0.02%     
==========================================
  Files         231      231              
  Lines       70613    70617       +4     
  Branches    70613    70617       +4     
==========================================
- Hits        55031    55026       -5     
- Misses      12462    12466       +4     
- Partials     3120     3125       +5     

Flag	Coverage Δ
unittests	77.92% <85.29%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wjones127]

Why are we passing down the legacy batch size here, instead of batch_size?

[westonpace]

DeletionRestorer uses the presence of this field to trigger a "v1 compatibility mode" if this is set (to match row group sizes). Otherwise the deletion restorer doesn't need to know the batch size.

I renamed the internal variable to legacy_batch_size as well to emphasize this. However, maybe I should rename to row_group_size or something better?

[wjones127]

I see. That's fine as is. Thanks for the explanation.