INT64 support for Delta Encoding

[asfimport]

As of now, parquet doesn't support delta encoding for INT64. However it is planned in the format:
https://github.com/Parquet/parquet-format/blob/master/Encodings.md

The benefits of this feature are huge. For timestamps it achieves twice better compression than SNAPPY on plain encoding, and the reading is faster. This feature is actually advertised on the home page of Parquet, even though it is not yet implemented:
http://parquet.incubator.apache.org/
http://image.slidesharecdn.com/hadoopsummit-140630160016-phpapp01/95/efficient-data-storage-for-analytics-with-apache-parquet-20-30-638.jpg?cb=1404162126

Reporter: Vassil Lunchev
Assignee: Vassil Lunchev

Related issues:

• Release Parquet-mr 1.9.0 (blocks)

PRs and other links:

• PR #154

_{Note: This issue was originally created as PARQUET-225. Please see the migration documentation for further details.}

[asfimport]

Vassil Lunchev:
Is somebody working on implementing this feature? I have a working code + unit tests for it. Is that interesting to you?

[asfimport]

Ryan Blue / @rdblue:
[~lunchev], I don't know of anyone working on it. Please open a pull request!

[asfimport]

Vassil Lunchev:
The Travis CI build passed for the pull request. Waiting for a review:
https://github.com/apache/incubator-parquet-mr/pull/154

[asfimport]

Jakub Liska:
Guys how do I enable it? I tried to encode :

Types.required(PrimitiveTypeName.INT64).named("foo")

that starts from 0 being incremented by 1, but I the resulting parquet file size is identical in 1.9.0 and 1.8.0 versions...

Is there something I have to enable for Delta Encoding to get applied?

[asfimport]

Ryan Blue / @rdblue:
[~l154k], parquet format 2.0 encodings aren't enabled by default because we haven't finalized the set of encodings that must be supported for 2.0. You can read and write the current set by explicitly setting your writer to use version 2.

[asfimport]

Jakub Liska:
@rdblue, I set it to version 2, I reported it here https://issues.apache.org/jira/browse/PARQUET-796

ParquetWriter writer = new ParquetWriter<Group>(outFile, new GroupWriteSupport, codec, blockSize, pageSize, dictPageSize, true, true, ParquetProperties.WriterVersion.PARQUET_2_0, configuration)