Implement RecordBatch::slice() to slice RecordBatches #460
Labels
arrow
Changes to the arrow crate
enhancement
Any new improvement worthy of a entry in the changelog
good first issue
Good for newcomers
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
When implementing an operator that needed to check for partitions across
RecordBatch
boundaries (in https://github.com/influxdata/influxdb_iox/pull/1733) I found myself wanting toslice
the record batch so I could keep part of it while I sent the rest throughIn addition I can imagine an operator like
RepartitionExec
which might want to break large batches into smaller batches when repartitioning.Describe the solution you'd like
Implement the following function
There is already a
Array::slice
method: https://github.com/apache/arrow-rs/blob/master/arrow/src/array/array.rs#L83-L98 so to slice aRecordBatch
you simply apply that function to each columnHere is a partial implementation, from https://github.com/influxdata/influxdb_iox/pull/1733
Describe alternatives you've considered
RecordBatch::split
is proposed in #343 but slice may be more general (and work better with existing Rust API ecosystem)Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: