Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid forced copy in Array::logical_nulls #5208

Open
alamb opened this issue Dec 14, 2023 · 0 comments
Open

Avoid forced copy in Array::logical_nulls #5208

alamb opened this issue Dec 14, 2023 · 0 comments
Labels
enhancement Any new improvement worthy of a entry in the changelog

Comments

@alamb
Copy link
Contributor

alamb commented Dec 14, 2023

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Some code requires "logical nullability" which is distinct from the "physical" nullability (what is defined by an Arrow arrays NullBuffer, if it has one)

To this end, arrow includes Array::logical_nulls which will compute logical nulls.

The issue is that it also potentially copies the NullBuffer for arrays that have it pre-computed, such as PrimitiveArray and StringArray. While the copy is perhaps not that bad (it is several Arcs) I would like to avoid it entirely if possible

Describe the solution you'd like
I would like to change logical_nulls to return a Cow to avoid the copy, as I propose in coralogix/arrow-datafusion#221

Describe alternatives you've considered
We can not make any changes

Additional context
Thus came up in DataFusion PR apache/datafusion#8511

Logical nulls was added in #4691

@alamb alamb added the enhancement Any new improvement worthy of a entry in the changelog label Dec 14, 2023
@alamb alamb changed the title Avoid a copy in Array::logical_nulls Avoid forced copy in Array::logical_nulls Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant