Parquet Memory Management in Java

[asfimport]

As discussed in HIVE-7685, Hive + Parquet often runs out memory when writing to many Hive partitions. This is quite problematic for our users.

Reporter: Brock Noland / @brockn
Assignee: Dong Chen / @dongc

Related issues:

• Warn when parquet memory manager kicks in (relates to)
• Parquet memory manager (is related to)

_{Note: This issue was originally created as PARQUET-108. Please see the migration documentation for further details.}

[asfimport]

Brock Noland / @brockn:
Dong, I have assigned this task to you because you are working on the Hive side of things. If I incorrectly assigned this to you, please let me know.

[asfimport]

Dong Chen / @dongc:
Hi Brock, it is assigned correctly. When working on the Hive side, I found the writers' dynamic buffer size and flush method is needed to manage their memory usage balance. These info can only be accessed in Parquet, so I will try to implement it here.

[asfimport]

Dong Chen / @dongc:
Hi @brockn, I uploaded a patch to implement the memory manager. Could you please help to review it?
The pull request is:
https://github.com/apache/incubator-parquet-mr/pull/80

[asfimport]

Brock Noland / @brockn:
Issue resolved by pull request 80
https://github.com/apache/incubator-parquet-mr/pull/80

[asfimport]

Abhilash L L:
Hello,

We are creating parquet files from within our MR code as one of the MultipleOutputs. We observer that it starts giving OOM when the number of partitions exceeded. We are using http://mvnrepository.com/artifact/com.twitter/parquet-avro/1.6.0

Will this work in our flow and not necessarily being written via hive ?

PS: Sorry for adding a comment on a resolved ticket