Improvements in ByteBuffer read path #1534
Description
For Apache Drill, we are looking to pass in a buffer that we have already allocated (in this case from Direct memory), wrapped in a ByteBuffer.
The current effort to allow a ByteBuffer read path is great except that the interface allocates the memory and there is no way for an application to pass in memory that has been allocated or (even better) to provide an allocator.
Additionally, it would be great to be able to use the same approach while decompressing.
As a starting point here is a patch on top of the ByteBuffer read effort that adds a function in CompatibilityUtils and also adds a ByteBuffer path for the Snappy Decompressor. The latter requires Hadoop 2.3 though, so some discussion on this would be called for.
Please let me have any feedback and I can make changes/additions.
Reporter: Parth Chandra / @parthchandra
Assignee: Jason Altekruse / @jaltekruse
Related issues:
- Binary column statistics error when reuse byte[] among rows (contains)
- ColumnChunkPageWriter uses only heap memory. (relates to)
- Move off of Parquet fork (is depended upon by)
PRs and other links:
Note: This issue was originally created as PARQUET-77. Please see the migration documentation for further details.