Encoding issue with fixed length byte arrays

[asfimport]

While running some tests against the master branch I hit an encoding issue that seemed like a bug to me.

I noticed that when writing a fixed length byte array and the array's size is > dictionaryPageSize (in my test it was 512), the encoding falls back to DELTA_BYTE_ARRAY as seen below:

Dec 17, 2014 3:41:10 PM INFO: parquet.hadoop.ColumnChunkPageWriteStore: written 12,125B for [flba_field] FIXED_LEN_BYTE_ARRAY: 5,000 values, 1,710B raw, 1,710B comp, 5 pages, encodings: [DELTA_BYTE_ARRAY]

But then read fails with the following exception:

Caused by: parquet.io.ParquetDecodingException: Encoding DELTA_BYTE_ARRAY is only supported for type BINARY
	at parquet.column.Encoding$7.getValuesReader(Encoding.java:193)
	at parquet.column.impl.ColumnReaderImpl.initDataReader(ColumnReaderImpl.java:534)
	at parquet.column.impl.ColumnReaderImpl.readPageV2(ColumnReaderImpl.java:574)
	at parquet.column.impl.ColumnReaderImpl.access$400(ColumnReaderImpl.java:54)
	at parquet.column.impl.ColumnReaderImpl$3.visit(ColumnReaderImpl.java:518)
	at parquet.column.impl.ColumnReaderImpl$3.visit(ColumnReaderImpl.java:510)
	at parquet.column.page.DataPageV2.accept(DataPageV2.java:123)
	at parquet.column.impl.ColumnReaderImpl.readPage(ColumnReaderImpl.java:510)
	at parquet.column.impl.ColumnReaderImpl.checkRead(ColumnReaderImpl.java:502)
	at parquet.column.impl.ColumnReaderImpl.consume(ColumnReaderImpl.java:604)
	at parquet.column.impl.ColumnReaderImpl.<init>(ColumnReaderImpl.java:348)
	at parquet.column.impl.ColumnReadStoreImpl.newMemColumnReader(ColumnReadStoreImpl.java:63)
	at parquet.column.impl.ColumnReadStoreImpl.getColumnReader(ColumnReadStoreImpl.java:58)
	at parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:267)
	at parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:131)
	at parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:96)
	at parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:136)
	at parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:96)
	at parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:129)
	at parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:198)
	... 16 more

When the array's size is < dictionaryPageSize, RLE_DICTIONARY encoding is used and read works fine:

Dec 17, 2014 3:39:50 PM INFO: parquet.hadoop.ColumnChunkPageWriteStore: written 50B for [flba_field] FIXED_LEN_BYTE_ARRAY: 5,000 values, 3B raw, 3B comp, 1 pages, encodings: [RLE_DICTIONARY, PLAIN], dic { 1 entries, 8B raw, 1B comp}

Reporter: Nezih Yigitbasi / @nezihyigitbasi
Assignee: Sergio Peña / @spena

Related issues:

• [Format] Encoding spec incorrect for DELTA_BYTE_ARRAY (causes)

PRs and other links:

• PR-225

_{Note: This issue was originally created as PARQUET-152. Please see the migration documentation for further details.}

[asfimport]

Ryan Blue / @rdblue:
I think the RLE_DICTIONARY behavior is probably because the dictionary is using plain encoding rather than delta byte array.

[asfimport]

Sergio Peña / @spena:
A fixed length byte array is still written as a BINARY, isn't it? If so, then I think we should allow FIXED_LEN_BYTE_ARRAY to be decoded by Encoding.DELTA_BYTE_ARRAY.

I tested something like this, and it worked:

DELTA_BYTE_ARRAY {
    @Override
    public ValuesReader getValuesReader(ColumnDescriptor descriptor,
        ValuesType valuesType) {
      if (descriptor.getType() != BINARY && descriptor.getType() != FIXED_LEN_BYTE_ARRAY) {
        throw new ParquetDecodingException("Encoding DELTA_BYTE_ARRAY is only supported for type BINARY and FIXED_LEN_BYTE_ARRAY");
      }
      return new DeltaByteArrayReader();
    }
  },

I'll create a PR, and run more tests to check this scenario.

[asfimport]

Sergio Peña / @spena:
Here's the PR for this bug:
#225

[asfimport]

Ryan Blue / @rdblue:
Thanks for fixing this, @spena!

[asfimport]

Antoine Pitrou / @pitrou:
It would be nice if the encodings spec had been updated as well, because for now it mentions that DELTA_BYTE_ARRAY is only supported for BYTE_ARRAY columns, not FIXED_LEN_BYTE_ARRAY. See PARQUET-2231.