Release parquet-mr 1.6.0

[asfimport]

Need to determine a list of tasks that should be done before release. Please add issues as sub-tasks.

Reporter: Ryan Blue / @rdblue
Assignee: Ryan Blue / @rdblue

Related issues:

• Provide callback to allow user defined key-value metadata merging strategy (blocks)
• Fix compatibility of parquet.metadata with 1.5.0 (is blocked by)
• Check parquet-scrooge compatibility (is blocked by)
• AvroReadSupport.setAvroDataSupplier is broken (is blocked by)
• Avro: Regression caused by schema handling (is blocked by)
• Restore ParquetInputSplit methods from 1.5.0 (is blocked by)
• Check ParquetWriter constructor compatibility with 1.5.0 (is blocked by)
• Make AvroParquetReader#builder() static (is blocked by)
• A benchmark module for Parquet would be nice (is blocked by)
• Add build instructions to the README (is blocked by)
• Update parquet version to 1.6.0 (relates to)

Original Issue Attachments:

• semver-diff.sh

_{Note: This issue was originally created as PARQUET-211. Please see the migration documentation for further details.}

[asfimport]

Ryan Blue / @rdblue:
I've been going through the command-line semver tool's reports for changes between 1.5.0 and the current master. There are lots of breaking changes in parquet.column, but I think that entire package is internal or SPI rather than API. I've flagged the following to fix:

• ParquetInputSplit#getExtraMetadata, #getFileSchema, #getRequestedSchema, #getBlocks, and #getReadSupportMetadata were removed and should be added back (This is now documented internal, but was part of the API and had external users).
• ParquetWriter constructors may have an incompatible change

And someone needs to look into this one:

• ParquetScroogeScheme#sink and #isSink were removed. (ScroogeStructConverter had removals, but I consider it internal)

Lastly, there are quite a few incompatible changes to parquet.metadata (see below). Is this public? It seems like it is part of the API because the metadata is exposed. Fixing it will be annoying because ColumnPath was removed entirely.

Class parquet.hadoop.metadata.Canonicalizer
 Removed Class , access public super synchronized
Class parquet.hadoop.metadata.ColumnChunkMetaData
 Added Method getPath, desc ()Lparquet/common/schema/ColumnPath;, access public
 Removed Method get, sig (Lparquet/hadoop/metadata/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set<Lparquet/column/Encoding;>;Lparquet/column/statistics/Statistics;JJJJJ)Lparquet/hadoop/metadata/ColumnChunkMetaData;, desc (Lparquet/hadoop/metadata/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set;Lparquet/column/statistics/Statistics;JJJJJ)Lparquet/hadoop/metadata/ColumnChunkMetaData;, access public static
 Added Method get, sig (Lparquet/common/schema/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set<Lparquet/column/Encoding;>;JJJJJ)Lparquet/hadoop/metadata/ColumnChunkMetaData;, desc (Lparquet/common/schema/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set;JJJJJ)Lparquet/hadoop/metadata/ColumnChunkMetaData;, access public static
 Added Method get, sig (Lparquet/common/schema/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set<Lparquet/column/Encoding;>;Lparquet/column/statistics/Statistics;JJJJJ)Lparquet/hadoop/metadata/ColumnChunkMetaData;, desc (Lparquet/common/schema/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set;Lparquet/column/statistics/Statistics;JJJJJ)Lparquet/hadoop/metadata/ColumnChunkMetaData;, access public static
 Removed Method getPath, desc ()Lparquet/hadoop/metadata/ColumnPath;, access public
 Removed Method get, sig (Lparquet/hadoop/metadata/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set<Lparquet/column/Encoding;>;JJJJJ)Lparquet/hadoop/metadata/ColumnChunkMetaData;, desc (Lparquet/hadoop/metadata/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set;JJJJJ)Lparquet/hadoop/metadata/ColumnChunkMetaData;, access public static
Class parquet.hadoop.metadata.ColumnChunkProperties
 Added Method getPath, desc ()Lparquet/common/schema/ColumnPath;, access public
 Removed Method getPath, desc ()Lparquet/hadoop/metadata/ColumnPath;, access public
 Removed Method get, sig (Lparquet/hadoop/metadata/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set<Lparquet/column/Encoding;>;)Lparquet/hadoop/metadata/ColumnChunkProperties;, desc (Lparquet/hadoop/metadata/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set;)Lparquet/hadoop/metadata/ColumnChunkProperties;, access public static
 Added Method get, sig (Lparquet/common/schema/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set<Lparquet/column/Encoding;>;)Lparquet/hadoop/metadata/ColumnChunkProperties;, desc (Lparquet/common/schema/ColumnPath;Lparquet/schema/PrimitiveType$PrimitiveTypeName;Lparquet/hadoop/metadata/CompressionCodecName;Ljava/util/Set;)Lparquet/hadoop/metadata/ColumnChunkProperties;, access public static
Class parquet.hadoop.metadata.ColumnPath
 Removed Class , access final public super synchronized

[asfimport]

Ryan Blue / @rdblue:
I'm attaching the wrapper script for semver I used to show all the changes.

[asfimport]

Ryan Blue / @rdblue:
I opened issues for the incompatibilities above and marked them as blockers.

• I'll fix PARQUET-234 and PARQUET-237
• PARQUET-220 and PARQUET-230 need reviews
• PARQUET-235 needs to be discussed (part of the API?)
• Someone needs to claim PARQUET-236

[asfimport]

Ryan Blue / @rdblue:
The IPMC release vote just passed, so 1.6.0 is officially released. I've updated the release SVN and it should propagate to mirrors in the next day or so.