Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Fixed reading arrays from parquet with required children #1140

Merged
merged 1 commit into from
Jul 6, 2022

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Jul 5, 2022

Some structs in arrow whose slots are nulls still require the underlying children to be populated (such as Struct and FixedSizeList). The current reading design does not support this.

This PR fixes this.

Closes #937

@jorgecarleitao jorgecarleitao added the bug Something isn't working label Jul 5, 2022
@jorgecarleitao jorgecarleitao changed the title Fixed reading arrow arrays from parquet with required values Fixed reading arrays from parquet with required children Jul 5, 2022
@codecov
Copy link

codecov bot commented Jul 5, 2022

Codecov Report

Merging #1140 (793a1bf) into main (78a2a63) will increase coverage by 0.05%.
The diff coverage is 78.91%.

@@            Coverage Diff             @@
##             main    #1140      +/-   ##
==========================================
+ Coverage   83.57%   83.62%   +0.05%     
==========================================
  Files         366      366              
  Lines       35941    35901      -40     
==========================================
- Hits        30038    30024      -14     
+ Misses       5903     5877      -26     
Impacted Files Coverage Δ
src/io/parquet/read/deserialize/primitive/basic.rs 95.60% <ø> (ø)
src/io/parquet/read/deserialize/utils.rs 78.46% <ø> (ø)
src/io/parquet/read/deserialize/boolean/nested.rs 71.42% <55.00%> (+4.36%) ⬆️
src/io/parquet/read/deserialize/binary/nested.rs 75.86% <80.00%> (+0.64%) ⬆️
src/io/parquet/read/deserialize/nested_utils.rs 82.89% <81.39%> (-0.75%) ⬇️
...rc/io/parquet/read/deserialize/primitive/nested.rs 87.71% <86.66%> (-0.52%) ⬇️
src/io/ipc/read/schema.rs 94.88% <0.00%> (+0.31%) ⬆️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78a2a63...793a1bf. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit 98e4913 into main Jul 6, 2022
@jorgecarleitao jorgecarleitao deleted the fix_struct branch July 6, 2022 01:44
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug reading parquet file with struct nested in list
1 participant