Support new min-max statistics in parquet-mr

[asfimport]

Impala started using new min-max statistics that got specified as part of PARQUET-686. Support for these should be added to parquet-mr as well.

Reporter: Zoltan Ivanfi / @zivanfi
Assignee: Gabor Szadovszky / @gszadovszky

Related issues:

• Logical-type-based toString for proper representeation in tools/logs (is depended upon by)

PRs and other links:

• PR - Option 1

_{Note: This issue was originally created as PARQUET-1025. Please see the migration documentation for further details.}

[asfimport]

Lars Volker / @lekv:
@zivanfi - Can you give a brief update on this effort? Are you actively working on it?

[asfimport]

Zoltan Ivanfi / @zivanfi:
Ah, no, sorry. I started working on in June/July, but then Impala's move to column indices made min/max stats seem deprecated and I abandoned the effort. Only later did I learn that column indices only make page-level min/max stats obsolete and min/max stats for whole row groups will still be useful, but at that time I was already working on a different task.

[asfimport]

Gabor Szadovszky / @gszadovszky:
To implement the new statistics we have to support the different comparison logics required by the specs. Currently, all the primitives are comparable. There are two possible options to extend the API:

1. Implement separate comparators which are to be used by parquet-mr internally as well as by the API users.
   • pros:
     • Backwards compatible (however, the statistics and filtering would work based on different ordering than before)
     • Keeps the primitive and logical types more loosely coupled in parquet-mr
   • cons:
     • It can be quite confusing to the API user that the compareTo methods of the provided primitive types shall not be used but the provided comparators
     • More parts of the API have to be modified to ensure that the proper comparators are used and these comparators are accessible
     • The client developers shall also modify their code parts where primitive comparison is used
2. Extend the actual primitive implementations for the logical types so the comparable objects would do the proper comparison by default.
   • pros:
     • Backwards compatible (however, the different Binary implementations would have different orderings than before, but the current one is incorrect anyway)
     • The API is more clean as the API users can rely on the comparable primitive types
     • The other parts of the API (e.g. filtering, statistics) can be kept unmodified, we modify only the parts where the primitives are created
     • The client code can be remain unmodified as it can still rely on the comparable primitive types
   • cons:

     • Proper comparison logic for UINT types will not be implemented (we cannot override the natural ordering of the primitive java types int and long)

     • The primitive and logical types would get more tightly coupled in parquet-mr

       Which one shall we prefer? I’m also curious about the ideas/comments of the API users (e.g. Hive, Sparks etc.).
       The first option is more or less implemented. Check the linked PR for details. I’m happy to implement the second option if it is more supported.

[asfimport]

Gabor Szadovszky / @gszadovszky:
On the 11/22 Parquet Sync meeting we have decided to go on with Option 1.
As a separate effort we shall implement proper compareTo methods for Binary types as far as practically feasible. (In the next major release 2.0 we will remove the Comparable logic from Binary.

[asfimport]

Ryan Blue / @rdblue:
Merged #425. Thanks for working on this, @gszadovszky!