Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Fixed panic in deserializing nested statistics #1139

Merged
merged 1 commit into from
Jul 5, 2022
Merged

Conversation

jorgecarleitao
Copy link
Owner

When deserializing nested statistics from parquet, we need to take into account that the null count of a List[Struct] can't be UInt64 and should instead be something like List[{"a": null_a, "b": null_b}]. This PR fixes this.

I think we can still improve this a bit (e.g. why not {"a": null_a, "b": null_b} instead?) so that users can use nested statistics to create predicates for filter pushdown of nested types, but at least this no longer panics.

@jorgecarleitao jorgecarleitao added the bug Something isn't working label Jul 4, 2022
@codecov
Copy link

codecov bot commented Jul 4, 2022

Codecov Report

Merging #1139 (d67dc12) into main (b3583b6) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1139      +/-   ##
==========================================
+ Coverage   83.49%   83.51%   +0.02%     
==========================================
  Files         366      366              
  Lines       35635    35691      +56     
==========================================
+ Hits        29752    29808      +56     
  Misses       5883     5883              
Impacted Files Coverage Δ
src/io/parquet/read/statistics/mod.rs 93.68% <100.00%> (+1.03%) ⬆️
src/io/ipc/read/reader.rs 95.84% <0.00%> (-0.70%) ⬇️
src/bitmap/immutable.rs 84.56% <0.00%> (+0.61%) ⬆️
src/bitmap/utils/slice_iterator.rs 98.78% <0.00%> (+1.21%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b3583b6...d67dc12. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit d87b38b into main Jul 5, 2022
@jorgecarleitao jorgecarleitao deleted the fix_list_stats branch July 5, 2022 15:19
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant