Bug reading parquet file with struct nested in list

[cjermain]

I discovered this bug while working with polars, but I have traced it down to the arrow2 code. It seems that a None value is not properly handled when a List-type Array has a Struct-type Array nested inside it. This is most easily reproduced by generating a parquet file as follows:

import pandas as pd

data = [                                                                                           
    {'a': [{'b': 0}]},
    {'a': [{'b': 1}]},
]

pd.DataFrame(data).to_parquet("~/test.parquet")  

The bug can be observed by adding a breakpoint for rust_panic in the following code that uses arrow2 to load the parquet file (this needs to be the most recent polars release - 0.13.21).

$ rust-gdb --args python -c "import polars; polars.read_parquet('~/test.parquet')"

I'm running into the error in the arrow2::io::parquet::read::deserialize::create_list function on line 46.

thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', /.../.cargo/git/checkouts/arrow2-9
45af624853845da/da703ae/src/io/parquet/read/deserialize/mod.rs:46:63
...
#8  0x00007ffff4c5cd0e in arrow2::io::parquet::read::deserialize::create_list (data_type=..., nested=0x7fffd05fa330, 
    values=...)
    at /.../.cargo/git/checkouts/arrow2-945af624853845da/da703ae/src/io/parquet/read/deserialize/mod.rs:46
46                  let (mut offsets, validity) = nested.nested.pop().unwrap().inner();
(gdb) l
41          nested: &mut NestedState,
42          values: Arc<dyn Array>,
43      ) -> Result<Arc<dyn Array>> {
44          Ok(match data_type {
45              DataType::List(_) => {
46                  let (mut offsets, validity) = nested.nested.pop().unwrap().inner();
47                  offsets.push(values.len() as i64);
48
49                  let offsets = offsets.iter().map(|x| *x as i32).collect::<Vec<_>>();
50                  Arc::new(ListArray::<i32>::new(

That is as far as I've had time to investigate so far. Outside of that issue I've seen nested structs working fine for the test cases I've looked at.

@jorgecarleitao, @ritchie46, what do you think?

[jorgecarleitao]

I agree that it is a bug ^^

[cjermain]

This is happening because of a remaining TODO in struct_.rs.

arrow2/src/io/parquet/read/deserialize/struct_.rs

Line 41 in 8fb3b8d

NestedState::new(vec![]), // todo

[scimas]

Possibly related,

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /path/to/arrow2-0.11.2/src/io/parquet/read/deserialize/mod.rs:272:44

The line in question, with context

arrow2/src/io/parquet/read/deserialize/mod.rs

Lines 271 to 273 in d035964

	columns_to_iter_recursive(
	vec![columns.pop().unwrap()],
	vec![types.pop().unwrap()],

[jorgecarleitao]

I will prioritize this.

[cjermain]

Thanks @jorgecarleitao!