Release Parquet format 2.5.0

[asfimport]

Reporter: Gabor Szadovszky / @gszadovszky
Assignee: Gabor Szadovszky / @gszadovszky

Related issues:

• parquet.thrift refers to wrong releases for the new compressions (is blocked by)
• Clarify ambiguous min/max stats for FLOAT/DOUBLE (is blocked by)

_{Note: This issue was originally created as PARQUET-1234. Please see the migration documentation for further details.}

[asfimport]

Ryan Blue / @rdblue:
Are we going to release a 2.4.1 with the changes for column index structures? I'd rather not wait on a resolution to PARQUET-1222 to get that out.

[asfimport]

Gidon Gershinsky / @ggershinsky:
Would it make sense to include the encryption PRs in parquet-format 2.5.0?

There are two related jiras  - PARQUET-1227 and PARQUET-1228. PR for the former is already  posted, and PR for the latter will follow soon after.

Having them in 2.5.0 will assist in integrating subsequent changes in parquet-mr.

[asfimport]

Zoltan Ivanfi / @zivanfi:
@rdblue, I suggest we decide this on the Parquet sync next week.

I hope PARQUET-1222 will not take that long to resolve as I already have a proposal in the JIRA description and I would like to create a PR early next week. Then we can discuss it in the Parquet sync and if everyone agrees to the proposal, we may resolve that issue by end of next week.

If, on the other hand, the community will raise problems, concerns, or new alternatives, the fix will take more time, and we may consider an earlier release as well. However, even if we don't introduce a new ordering in the next release, we should still clarify the specification by explicitly calling out this ambiguity and the countermeasures that have to be taken to deal with the consequences (basically the bullet points at the end of the description of PARQUET-1222). And I think this is a significant change in itself that may deserve a minor version change.

So we do not necessarily have to block the release on a proper solution for PARQUET-1222 if we expect it to take a long time, but in my opinion we should at least address that issue partially by blocking on a new JIRA that is a subset of PARQUET-1222.

[asfimport]

Zoltan Ivanfi / @zivanfi:
@ggershinsky, I consider PARQUET-1222 to be serious enough to justify an urgent release with a fix. Personally, I would rather not delay this by waiting for a new feature, but ultimately it is the community that decides this.

[asfimport]

Gidon Gershinsky / @ggershinsky:
@zivanfi, the PR for PARQUET-1227 is in, and the code for PARQUET-1228 is ready (I'll send a PR for it after the first PR is reviewed). So technically, this feature can be included in parquet-format any time before the end of next week, similar to PARQUET-1222. But, as you said, this is up to the community.

[asfimport]

Zoltan Ivanfi / @zivanfi:
@ggershinsky, some PRs can spend several weeks in the review phase. Of course, if it is accepted quickly, it will be included in the release.

[asfimport]

Gidon Gershinsky / @ggershinsky:
@zivanfi, I see, thanks. I will hope for a faster timeline :)