-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Closed
Milestone
Description
The sorting order of the floating point values are not properly specified, therefore NaN values can cause skipping valid values when filtering. See PARQUET-1222 for more info.
This issue is for ignoring statistics for float/double if it contains NaN to prevent data loss at the read path when filtering.
Reporter: Gabor Szadovszky / @gszadovszky
Assignee: Gabor Szadovszky / @gszadovszky
Related issues:
- Release Parquet Java 1.10 (blocks)
- Release Parquet-mr 1.8.3 (blocks)
- Specify a well-defined sorting order for float and double types (relates to)
- PPD: Floating point NaN is not transitive across comparisons (relates to)
PRs and other links:
Note: This issue was originally created as PARQUET-1246. Please see the migration documentation for further details.