ArrayIndexOutOfBoundsException with Parquet write version v2

[asfimport]

I am getting the following exception when reading a parquet file that was created using Avro WriteSupport and Parquet write version v2.0:

Caused by: parquet.io.ParquetDecodingException: Can't read value in column [colName, rows, array, name] BINARY at value 313601 out of 428260, 1 out of 39200 in currentPage. repetition level: 0, definition level: 2
	at parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:462)
	at parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:364)
	at parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:405)
	at parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:209)
	... 27 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
	at parquet.column.values.deltastrings.DeltaByteArrayReader.readBytes(DeltaByteArrayReader.java:70)
	at parquet.column.impl.ColumnReaderImpl$2$6.read(ColumnReaderImpl.java:307)
	at parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:458)
	... 30 more

The file is quite big (500Mb) so I cannot upload it here, but possibly there is enough information in the exception message to understand the cause of error.

Reporter: Konstantin Shaposhnikov / @kostya-sh
Assignee: Konstantin Shaposhnikov / @kostya-sh

Related issues:

• Release Parquet 1.8.0 (blocks)
• DeltaByteArrayReader fails with ArrayIndexOutOfBoundsException when moving across pages (duplicates)

PRs and other links:

• PR #235: Correctly read data written with this bug.
• PR #171: Write-side fix for this bug.

_{Note: This issue was originally created as PARQUET-246. Please see the migration documentation for further details.}

[asfimport]

Konstantin Shaposhnikov / @kostya-sh:
When the exception occurred in DeltaByteArrayReader.readBytes method:

• previous = Binary{0 bytes, []}
• prefixLength = 1

Could it be because the same DeltaByteArrayWriter was reused across multiple pages?

[asfimport]

Konstantin Shaposhnikov / @kostya-sh:
The following pull request seems to fix the problem: https://github.com/apache/incubator-parquet-mr/pull/171

The following code can be used to generate problematic parquet file:

import java.io.File;

import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;

import parquet.column.ParquetProperties.WriterVersion;
import parquet.hadoop.ParquetWriter;
import parquet.hadoop.api.WriteSupport;
import parquet.hadoop.metadata.CompressionCodecName;

public class Test {
	  @SuppressWarnings("unchecked")
	  private static <T> WriteSupport<T> writeSupport(Schema avroSchema) {
	    return (WriteSupport<T>) new AvroWriteSupport(
	        new AvroSchemaConverter().convert(avroSchema), avroSchema);
	  }
	
	public static void main(String[] args) throws Exception {
		Schema schema = new Schema.Parser().parse("{\"type\":\"record\",\"name\":\"A\",\"namespace\":\"B\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}");
		
		new File("C:/Temp/test.parquet").delete();
		Path outputPath = new Path("C:/Temp/test.parquet");
		
		ParquetWriter<GenericRecord> parquetOut = new ParquetWriter<GenericRecord>(outputPath,
				Test.<GenericRecord>writeSupport(schema),
				CompressionCodecName.SNAPPY,
				ParquetWriter.DEFAULT_BLOCK_SIZE, ParquetWriter.DEFAULT_PAGE_SIZE, ParquetWriter.DEFAULT_PAGE_SIZE,
				true, // enable dictionary
				false, // validating
				WriterVersion.PARQUET_2_0,
				new Configuration());
		
		for (int i = 0; i < 100000; i++) {
			GenericRecord record = new GenericData.Record(schema);
			record.put("id", "a" + i);
			parquetOut.write(record);
		}

		parquetOut.close();
	}
}

When dumping the created file using dump command from parquet tools an exception is shown.

[asfimport]

Alex Levenson / @isnotinvain:
Issue resolved by pull request 171
#171

[asfimport]

Ryan Blue / @rdblue:
Should we also update the read side so we can recover data written with this bug? Does this happen when reading the entire file, or just when reading from a middle row group in MR?

[asfimport]

Michael MacFadden / @mallman:
@rdblue Yes, a read-side fix would be very helpful so we can read files corrupted by this bug.

[asfimport]

Ryan Blue / @rdblue:
@mallman can you answer my questions about this? When does this happen? Whenever you read a file like this? If so, then we need to add support (with a flag) to initialize the delta byte array from the last value in the last page/row group. That would mean we also need to keep it around and throw an exception if it isn't present (if you were reading from the middle of the file, we can't back up to get it right). I think data recovery needs to be part of the solution for this.

[asfimport]

Michael MacFadden / @mallman:
Hi @rdblue. Sorry, I'm still wrapping my head around this problem. I'm just at the point where I have a parquet file I can't read with the parquet-dump command. This is what I'm seeing so far:

[msa@50-204-120-228-static chunk=1434499205000-1434499920000]$ parquet-dump -m --debug part-r-00001.parquet > part-r-00001.parquet.dump.rc7
parquet.io.ParquetDecodingException: Can't read value in column [response, seatbid, array, bid, array, adm] BINARY at value 201 out of 8794, 1 out of 200 in currentPage. repetition level: 0, definition level: 6
	at parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:462)
	at parquet.column.impl.ColumnReaderImpl.getBinary(ColumnReaderImpl.java:410)
	at parquet.tools.command.DumpCommand.dump(DumpCommand.java:291)
	at parquet.tools.command.DumpCommand.dump(DumpCommand.java:218)
	at parquet.tools.command.DumpCommand.execute(DumpCommand.java:139)
	at parquet.tools.Main.main(Main.java:222)
Caused by: java.lang.ArrayIndexOutOfBoundsException
	at parquet.column.values.deltastrings.DeltaByteArrayReader.readBytes(DeltaByteArrayReader.java:70)
	at parquet.column.impl.ColumnReaderImpl$2$6.read(ColumnReaderImpl.java:307)
	at parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:458)
	... 5 more
Can't read value in column [response, seatbid, array, bid, array, adm] BINARY at value 201 out of 8794, 1 out of 200 in currentPage. repetition level: 0, definition level: 6

This parquet file has a single row group. I haven't yet determined whether this is affecting all of the files we have or just some. LMK if there's anything else about this file you'd like me to send, like a metadata dump. Alternatively, I believe I can send you the file itself if you'd like to see it. I would have to double check from the higher ups. The file itself is only 3.9 MB. I could put it in a dropbox or something.

Cheers.

[asfimport]

Sergio Peña / @spena:
I have the same error using one of the files generated by parquet-benchmarks.

First, parquet-benchmark read fails due to the PARQUET-152 issue. I skipped PARQUET-152 issue by writing the new files with a dictionary page size equals to the byte array size. After that, the read fails with the same error on this ticket:

org.apache.parquet.io.ParquetDecodingException: Can not read value at 188801 in block 0 in file file:/opt/local/parquet/upstream/parquet-mr/parquet-benchmarks/target/tests/ParquetBenchmarks/PARQUET-1M-BS256M_PS4M
        at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:240)
        at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:124)
        at org.apache.parquet.benchmarks.ReadBenchmarks.read(ReadBenchmarks.java:40)
        at org.apache.parquet.benchmarks.ReadBenchmarks.read1MRowsBS256MPS4MUncompressed(ReadBenchmarks.java:64)
        at org.apache.parquet.benchmarks.generated.ReadBenchmarks_read1MRowsBS256MPS4MUncompressed.read1MRowsBS256MPS4MUncompressed_avgt_jmhStub(ReadBenchmarks_read1MRowsBS256MPS4MUncompressed.java:170)
        at org.apache.parquet.benchmarks.generated.ReadBenchmarks_read1MRowsBS256MPS4MUncompressed.read1MRowsBS256MPS4MUncompressed_AverageTime(ReadBenchmarks_read1MRowsBS256MPS4MUncompressed.java:134)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:452)
        at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:434)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.parquet.io.ParquetDecodingException: Can't read value in column [binary_field] BINARY at value 188801 out of 1000000, 1 out of 94400 in currentPage. repetition level: 0, definition level: 0
        at org.apache.parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:462)
        at org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:364)
        at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:405)
        at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:215)
        ... 15 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
        at org.apache.parquet.column.values.deltastrings.DeltaByteArrayReader.readBytes(DeltaByteArrayReader.java:70)
        at org.apache.parquet.column.impl.ColumnReaderImpl$2$6.read(ColumnReaderImpl.java:307)
        at org.apache.parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:458)
        ... 18 more

I'll help investigating this issue. I am working on updating some benchmark tests.

[asfimport]

Sergio Peña / @spena:
Here's a pull request to read files written with the issue.
#171

Please review. Seems it is working correctly.

[asfimport]

Michael MacFadden / @mallman:
@spena Thanks for helping with this. I'll give your patch a try in the morning.

BTW, I think you meant to link to #217.

[asfimport]

Michael MacFadden / @mallman:
@spena I tried your patch and was able to successfully dump my previously unreadable parquet file. Thanks!

[asfimport]

Ryan Blue / @rdblue:
Merged #235 to read data written with this bug. This will fail MR jobs unless "parquet.split.files" is set to false because data must be read sequentially from the start of the file.

[asfimport]

Ryan Blue / @rdblue:
Closing this now that the read side has a fix. Thanks Konstantin, Sergio, Alex, and Tianshuo for all your work getting this resolved!

[asfimport]

Michael MacFadden / @mallman:
Thanks everyone for your awesome work on this. @rdblue to make sure I understand correctly, will setting parquet.split.files=false read all parquet files sequentially or only the ones with the defective encoding?