PARQUET-1771: Support configurable for DirectByteBufferAllocator from Hadoop Configuration#749
PARQUET-1771: Support configurable for DirectByteBufferAllocator from Hadoop Configuration#749LiShuMing wants to merge 2 commits intoapache:masterfrom
Conversation
Fokko
left a comment
Fokko
left a comment
There was a problem hiding this comment.
We already have:
public Builder withAllocator(ByteBufferAllocator allocator) {
this.allocator = allocator;
return this;
}
which provides similar functionality. Or am I missing something.
| return this; | ||
| } | ||
|
|
||
| public Builder useDirectByteBufferAllocator(boolean val) { |
There was a problem hiding this comment.
| public Builder useDirectByteBufferAllocator(boolean val) { | |
| public Builder useDirectByteBufferAllocator(boolean useDirectByteBuffer) { |
There was a problem hiding this comment.
Thanks for replies.
In Spark, ParquetFileReader passes options from Hadoop Configuration (using ParquetReadOptions.Builder) rather than ParquetReadOptions.
In ParquetReadOptions.Builder constructor, now Allocator param cannot be passed to ParquetReadOptions.
There was a problem hiding this comment.
Makes sense, could you still rename val to useDirectByteBuffer?
There was a problem hiding this comment.
Hi~ I have fixed this and rebased the newest codes.
Can you review this again?
… Hadoop Configuration
|
I wonder, if it solves https://issues.apache.org/jira/browse/PARQUET-1006 too. Could you check it please? |
3 participants
Jira
Tests
Commits