-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve Nullable support during dataframe arithmetic operations #6825
Labels
enhancement
New feature or request
Comments
asmirnov82
changed the title
Improve Nullable support during dataframe arithmetic operations and avoid excessive cloning of the left side
Improve Nullable support during dataframe arithmetic operations
Sep 21, 2023
ghost
added
the
in-pr
label
Sep 29, 2023
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
During arithmetic operations dataframe performs cloning the left side column into the result to have validity bitmap and than checks the right side validity bitmap for NULL value.
For example for Multiply we do cloning in case of inPlace parameter is set to false (default behavior):
and inside container for each value we check validity:
Validity check is a very slow operation. It's possible to calculate Raw values and then use binary logic (AND) for calculating validity bitmap for whole byte.
The text was updated successfully, but these errors were encountered: