refactor(rust): Refactor compute kernels in polars-arrow to avoid using gather

[nameexhaustion]

Before we can move the gather logic to polars-compute we need to remove all uses of it in polars-arrow, as it will no longer be accessible in polars-arrow after the move.

[nameexhaustion]

Casting nullable List -> FixedSizeList, used a gather to ensure the width of the null slots - have updated this to use Growable instead.

[nameexhaustion]

list.get() / array.get() were building selection indices and then calling gather with them - I've re-written them to use loops instead.

[nameexhaustion]

drive-by - print the oob index in error message

[codecov]

Codecov Report

Attention: Patch coverage is 91.92547% with 13 lines in your changes missing coverage. Please review.

Project coverage is 79.73%. Comparing base (8335f75) to head (5dff00a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...polars-arrow/src/legacy/kernels/fixed_size_list.rs	87.37%	13 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #19669   +/-   ##
=======================================
  Coverage   79.72%   79.73%           
=======================================
  Files        1542     1542           
  Lines      212208   212232   +24     
  Branches     2449     2449           
=======================================
+ Hits       169182   169220   +38     
+ Misses      42472    42458   -14     
  Partials      554      554           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

There might be some performance implications in those rewrites. It's hard to tell. I think we actually should move the cast to polars-compute as well. Then the dependency problem is resolved.

[nameexhaustion]

There might be some performance implications in those rewrites. It's hard to tell. I think we actually should move the cast to polars-compute as well. Then the dependency problem is resolved.

I tried to move the casting code, but I don't think it's possible as the cast is currently used by ArrowArray::new() in the polars-arrow crate -

polars/crates/polars-arrow/src/ffi/array.rs

Lines 111 to 114 in 3cdb7c2

	let variadic_buffer_sizes = if needs_variadic_buffer_sizes {
	#[cfg(feature = "compute_cast")]
	{
	let arr = crate::compute::cast::cast_unchecked(

From benchmarking, the PR as it is improves list->array casting performance, while regressing on list.get() / array.get() performance -

# DF
shape: (20_000_000, 2)
┌──────────┬─────────────────────────────────┐
│ i64      ┆ list                            │
│ ---      ┆ ---                             │
│ i64      ┆ list[i64]                       │
╞══════════╪═════════════════════════════════╡
│ 6765403  ┆ [6765403, 6765403, … 6765403]   │
│ 16059030 ┆ [16059030, 16059030, … 1605903… │
...
# This PR
cast list->array 0.7143477080389857
list.get(i64) 0.2533301250077784
arr.get(col(indices)) 0.860774208791554
# 1.12.0
cast list->array 1.0722814579494298
list.get(i64) 0.2003235830925405
arr.get(col(indices)) 0.5676086670719087

I believe for list.get() / array.get(), switching to the growable introduced dynamic dispatch for every row, is the cause of the performance regression. On the other hand cast() performance improved as it was already dynamic dispatch, but we removed the extra step of materializing selection indices.

I think what I'll try next is to probably move list/array.get() to polars-compute as well together with the gather, which should work around the dependency issues.

[ritchie46]

I tried to move the casting code, but I don't think it's possible as the cast is currently used by ArrowArray::new() in the polars-arrow crate -

Oh, that one we can match the dtype (it's either BinaryView, or Utf8View) and as_any().downcast_ref(). Get that type out and then call to_binview if it is Utf8View.