-
Notifications
You must be signed in to change notification settings - Fork 485
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ORC-1813: [C++] Fix has_null forward compatibility #2082
Conversation
Thank you for making a PR, @suxiaogang223 . |
I have fix some orc c++ uts because the orc file |
Thanks for fixing this! Could you create a JIRA issue? We use JIRA instead of GitHub issue for tracking purpose. |
How to create jira? it seems that your jira system cannot be used :( |
You may request a JIRA account via: https://selfserve.apache.org/jira-account.html It was open for registration. Unfortunately it has been disabled due to spam :( |
Thank you for fling JIRA and update the PR title, @suxiaogang223 . BTW, does it affect all ORC versions like 1.9.x and 1.8.x? The JIRA issue seems to be reported on Apache ORC 2.0.x only. |
For now, I set the milestone, 2.0.4, based on |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thank you @suxiaogang223 @wgtmac @dongjoon-hyun ~
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
Thank you all, @suxiaogang223 , @wgtmac , @ffacs ! Merged to Could you make a backporting PR to |
close: apache#2079 relate pr: apache#2055 Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation. The Java implementation includes the following logic: ```java if (stats.hasHasNull()) { hasNull = stats.getHasNull(); } else { hasNull = true; } ``` In contrast, the C++ implementation directly uses the has_null value without any fallback logic: ```c++ ColumnStatisticsImpl::ColumnStatisticsImpl(const proto::ColumnStatistics& pb) { stats_.setNumberOfValues(pb.number_of_values()); stats_.setHasNull(pb.has_null()); } ``` We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly. **This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(** I have tested this using [Doris](https://github.com/apache/doris) external pipeline: apache/doris#45104 apache/doris-thirdparty#259 No Closes apache#2082 from suxiaogang223/fix_has_null. Authored-by: Socrates <suxiaogang223@icloud.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
There is a conflict fixed pr : #2086 |
close: #2079 relate pr: #2055 Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation. The Java implementation includes the following logic: ```java if (stats.hasHasNull()) { hasNull = stats.getHasNull(); } else { hasNull = true; } ``` In contrast, the C++ implementation directly uses the has_null value without any fallback logic: ```c++ ColumnStatisticsImpl::ColumnStatisticsImpl(const proto::ColumnStatistics& pb) { stats_.setNumberOfValues(pb.number_of_values()); stats_.setHasNull(pb.has_null()); } ``` We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly. **This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(** I have tested this using [Doris](https://github.com/apache/doris) external pipeline: apache/doris#45104 apache/doris-thirdparty#259 No Closes #2082 from suxiaogang223/fix_has_null. Authored-by: Socrates <suxiaogang223icloud.com> ### What changes were proposed in this pull request? ### Why are the changes needed? ### How was this patch tested? ### Was this patch authored or co-authored using generative AI tooling? Closes #2086 from suxiaogang223/cherry_pick_fix_has_null. Authored-by: Socrates <suxiaogang223@icloud.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
For the record, the following landed to branch-2.0. |
What changes were proposed in this pull request?
close: #2079
relate pr: #2055
Introduce fallback logic in the C++ reader to set hasNull to true when the field is missing, similar to the Java implementation.
The Java implementation includes the following logic:
In contrast, the C++ implementation directly uses the has_null value without any fallback logic:
Why are the changes needed?
We encountered an issue with the C++ implementation of the ORC reader when handling ORC files written with version 0.12. Specifically, files written in this version do not include the hasNull field in the column statistics metadata. While the Java implementation of the ORC reader handles this gracefully by defaulting hasNull to true when the field is absent, the C++ implementation does not handle this scenario correctly.
This issue prevents predicates like IS NULL from being pushed down to the ORC reader!!! As a result, all rows in the file are filtered out, leading to incorrect query results :(
How was this patch tested?
I have tested this using Doris external pipeline:
apache/doris#45104
apache/doris-thirdparty#259
Was this patch authored or co-authored using generative AI tooling?
No