feat: Enhance `array_slice` functionality to support `ListView` and `LargeListView` types #18432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

Weijun-H merged 10 commits into apache:main from Weijun-H:18351-array-slice-listview

Nov 8, 2025

Member

Weijun-H commented Nov 1, 2025 •

edited

Loading

Which issue does this PR close?

Closes Support slicing ListView #18351

Rationale for this change

array_slice accepts ListView / LargeListView inputs.

What changes are included in this PR?

Extend array_slice_inner to handle ListView/LargeListView arrays directly.
Share the stride/bounds logic between list and list‑view implementations via a new SlicePlan.

Are these changes tested?

Yes

Are there any user-facing changes?

Yes. array_slice now accepts ListView and LargeListView arrays without requiring an explicit cast.

Weijun-H marked this pull request as draft

November 1, 2025 19:26

github-actions bot added the common label

Member Author

Weijun-H commented Nov 1, 2025

Weijun-H changed the title ~~feat: Enhance array_slice functionality to support ListView and LargeListView types~~ feat: Enhance array_slice functionality to support ListView and LargeListView types

Weijun-H marked this pull request as ready for review

November 1, 2025 20:09

vegarsti approved these changes

View reviewed changes

Contributor

vegarsti left a comment •

edited

Loading

Very nice refactor! The SlicePlan and extraction of functions makes the rest of the code a lot more readable.

datafusion/functions-nested/src/extract.rs Outdated

Comment on lines 778 to 786

    
                  let field = match array.data_type() {

                      ListView(field) | LargeListView(field) => Arc::clone(field),

                      other => {

                          return Err(internal_datafusion_err!(

                              "array_slice got unexpected data type: {}",

                              other

                          ));

                      }

                  };

Contributor

vegarsti Nov 2, 2025

Suggest moving this to before the slice to avoid unnecessary work

Member Author

Weijun-H Nov 2, 2025

Nice catch!

datafusion/functions-nested/src/extract.rs

Contributor

vegarsti Nov 2, 2025 •

edited

Loading

Nice tests! This PR for reverse of FixedSizeList also added sqllogictest cases https://github.com/apache/datafusion/pull/16423/files#diff-317c67cc9ce87268e4ccec1cb75316eed82f99ae2ffc226874e5897913ffa4c8

It doesn't look like that is currently possible for ListView on arrow-rs 57, as it hits this path, which did not have a branch for ListView at the time of release
https://github.com/apache/arrow-rs/blob/062d766a9c3070d191d1a1fd0baca01b9d13994f/arrow-schema/src/datatype_parse.rs#L70-L96 (but it was added recently: apache/arrow-rs#8649)

Contributor

Jefffrey Nov 2, 2025

Perhaps we can add to SLTs and expect it to fail on the cast (leaving the expected results as comments) so when we bump to arrow with that fix we will automatically know

Contributor

vegarsti Nov 2, 2025

Good idea!

Member Author

Weijun-H Nov 2, 2025

Added

datafusion/functions-nested/src/extract.rs Show resolved Hide resolved

github-actions bot added the sqllogictest label

Weijun-H requested a review from Jefffrey

November 2, 2025 19:50

Jefffrey mentioned this pull request

Fast scalar path for array_slice #18458

Open

Jefffrey reviewed

View reviewed changes

Contributor

Jefffrey left a comment

I like the introduction of SlicePlan

datafusion/functions-nested/src/extract.rs

    
                  }

              }

              fn adjusted_from_index<O: OffsetSizeTrait>(index: i64, len: O) -> Result<Option<O>>

Contributor

Jefffrey Nov 3, 2025 •

edited

Loading

Looks like this + adjusted_to_index are pulled out from their inner function general_array_slice verbatim without changes 👍

(Just a note for myself so I know there isn't an actual diff on the function)

datafusion/functions-nested/src/extract.rs

Comment on lines +707 to +713

    
                  let field = match array.data_type() {

                      ListView(field) | LargeListView(field) => Arc::clone(field),

                      other => {

                          return internal_err!("array_slice got unexpected data type: {}", other);

                      }

                  };

Contributor

Jefffrey Nov 3, 2025

Suggested change

      
                let field = match array.data_type() {
          
                    ListView(field) | LargeListView(field) => Arc::clone(field),
          
                    other => {
          
                        return internal_err!("array_slice got unexpected data type: {}", other);
          
                    }
          
                };
          
                let field = match array.data_type() {
          
                    ListView(field) | LargeListView(field) => Arc::clone(field),
          
                    _ => unreachable!()
          
                };

Given array is most definitely a GenericListViewArray

Though I wonder why we handle this differently to how its done in general_array_slice 🤔

datafusion/functions-nested/src/extract.rs Show resolved Hide resolved

datafusion/functions-nested/src/extract.rs Show resolved Hide resolved

Weijun-H force-pushed the 18351-array-slice-listview branch from defabd2 to 9dee50f Compare

November 4, 2025 20:54

Jefffrey approved these changes

View reviewed changes

Contributor

Jefffrey left a comment

Should be good to go after fixing the CI checks

Weijun-H added 10 commits

November 8, 2025 11:39


          feat: Enhance array_slice functionality to support ListView and Large…

dfc7905

…ListView types


          refactor: Reduced duplication

597e041


          chore

6c2e9d6


          chore

06a87ea


          chore: Added tests for ooo and null

e22dc54


          chore

380069e


          chore: Add tests in slt

449f1ef


          fix: Supported null stride

851ffd0


          chore: fmt

020efa8


          chore

8ff4008

Weijun-H force-pushed the 18351-array-slice-listview branch from 59175d6 to 8ff4008 Compare

November 8, 2025 09:40

github-actions bot removed the common label

Weijun-H added this pull request to the merge queue

Merged via the queue into apache:main with commit 3f2b6eb

28 checks passed

Member Author

Weijun-H commented Nov 8, 2025

Thanks @vegarsti and @Jefffrey for reviewing 👍

hsiang-c pushed a commit to hsiang-c/datafusion that referenced this pull request


          feat: Enhance array_slice functionality to support ListView and `…

074e645

…LargeListView` types (apache#18432)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Closes apache#18351

## Rationale for this change

`array_slice` accepts `ListView` / `LargeListView` inputs.

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?


- Extend array_slice_inner to handle `ListView`/`LargeListView` arrays
directly.
- Share the stride/bounds logic between list and list‑view
implementations via a new `SlicePlan`.



<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?
Yes

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
Yes. `array_slice` now accepts `ListView` and `LargeListView` arrays
without requiring an explicit cast.

codetyri0n pushed a commit to codetyri0n/datafusion that referenced this pull request


          feat: Enhance array_slice functionality to support ListView and `…

787f5d3

…LargeListView` types (apache#18432)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Closes apache#18351

## Rationale for this change

`array_slice` accepts `ListView` / `LargeListView` inputs.

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?


- Extend array_slice_inner to handle `ListView`/`LargeListView` arrays
directly.
- Share the stride/bounds logic between list and list‑view
implementations via a new `SlicePlan`.



<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?
Yes

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
Yes. `array_slice` now accepts `ListView` and `LargeListView` arrays
without requiring an explicit cast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels