TwoLevelListWriter does not handle null values in array

[asfimport]

parquet-mr is unable to handle the following avro schema:

{"type": "record",
 "namespace": "com.cloudera.impala",
 "name": "table_3",
 "fields": [
   {"name": "field_6", "type":
     {"type": "array", "items": ["null",
       {"type": "map", "values": ["null", "string"]}]}}]}

If map is null, the following exception happens:

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at parquet.avro.AvroWriteSupport.writeMap(AvroWriteSupport.java:185)
	at parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:277)
	at parquet.avro.AvroWriteSupport.access$400(AvroWriteSupport.java:48)
	at parquet.avro.AvroWriteSupport$TwoLevelListWriter.writeCollection(AvroWriteSupport.java:473)
	at parquet.avro.AvroWriteSupport$ListWriter.writeList(AvroWriteSupport.java:322)
	at parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:275)
	at parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:169)
	at parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:144)
	at parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
	at parquet.hadoop.ParquetWriter.write(ParquetWriter.java:324)
	at com.cloudera.impala.datagenerator.RandomNestedDataGenerator.writeFile(RandomNestedDataGenerator.java:69)
	at com.cloudera.impala.datagenerator.RandomNestedDataGenerator.main(RandomNestedDataGenerator.java:284)

The cause is probably because if there is a null value in the array, the TwoLevelListWriter does not check if an element is null: https://github.com/apache/parquet-mr/blob/master/parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java#L456

Reporter: Taras Bobrovytsky
Assignee: Ryan Blue / @rdblue

Related issues:

• Release Parquet-mr 1.9.0 (blocks)

PRs and other links:

• PR #291

_{Note: This issue was originally created as PARQUET-387. Please see the migration documentation for further details.}

[asfimport]

Cheng Lian / @liancheng:
I believe setting parquet.avro.write-old-list-structure to false fixes this issue.

[asfimport]

Ryan Blue / @rdblue:
As Cheng notes, setting the configuration property to use the new, 3-level list structure fixes the problem. Support for arrays that contain null was the point of moving to the 3-level structure. Unfortunately, we couldn't do that by default because it is a behavior change. That's why we have the property.

As far as what the correct behavior is for this case, I think the NPE is actually right. Because the 2-level structure can't handle null values, we have the choice of either silently dropping them or throwing an exception. Silently dropping them clearly isn't a good idea, so the NPE is the best option. What we could do to make this more friendly is catch the NPE and throw a new one with an error message to tell you what happened: that null values aren't allowed in Avro arrays unless you write using the property.

[asfimport]

Ryan Blue / @rdblue:
I've fixed the error messages in PR #291. I think that's all we need to do for this.

[asfimport]

Ryan Blue / @rdblue:
Thanks for reviewing, [~lian cheng]! The error message for this case is now more friendly.