-
Notifications
You must be signed in to change notification settings - Fork 810
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
u64::MAX does not roundtrip through parquet #254
Comments
Some more context: the logical type
so in theory we could map the logical range aka: "reinterpret signed as unsigned ints". Then the only bit that might be tricky (or not in which case a to-be-written test will pass) is to get the statistics right. |
BTW: I'm working on that issue. |
- updates arrow to parquet type mapping to use reinterpret/overflow cast for u64<->i64 similar to what the C++ stack does - changes statistics calculation to account for the fact that u64 should be compared unsigned (as per spec) Fixes apache#254.
- updates arrow to parquet type mapping to use reinterpret/overflow cast for u64<->i64 similar to what the C++ stack does - changes statistics calculation to account for the fact that u64 should be compared unsigned (as per spec) Fixes apache#254.
- updates arrow to parquet type mapping to use reinterpret/overflow cast for u64<->i64 similar to what the C++ stack does - changes statistics calculation to account for the fact that u64 should be compared unsigned (as per spec) Fixes apache#254.
* re-export arity kernels in `arrow::compute` Seems logical since all other kernels are re-exported as well under this flat hierarchy. * return file from `parquet::arrow::arrow_writer::tests::[one_column]_roundtrip` * support full arrow u64 through parquet - updates arrow to parquet type mapping to use reinterpret/overflow cast for u64<->i64 similar to what the C++ stack does - changes statistics calculation to account for the fact that u64 should be compared unsigned (as per spec) Fixes #254. * avoid copying array when reading u64 from parquet * support full arrow u32 through parquet This is idential to the solution we now have for u64.
Describe the bug
u64::MAX
gets truncated to 0 when storing to parquet and reading back.To Reproduce
Add the following test:
Expected behavior
All values either roundtrip correctly or some error is produced - at least they should not be silently truncated.
Additional context
Tested on commit
8f030db53d9eda901c82db9daf94339fc447d0db
.The text was updated successfully, but these errors were encountered: