Core, Parquet: Add filter to InternalData #14630
Conversation
|
This is needed for a follow-up step for the partition stat Scan API to filter the stats. InternalData is used to read the partition stats and as a next step filtering is to be introduced, however, InternalData doesn't have a support for this now. |
|
Anther way to implement this is to introduce a SupportsFiltering interface and parquet reader can derive from that. At the creation site we have to check if the reader is an instance of SupportsFiltering and add filter if yes. |
| /** Set a custom class for in-memory objects at the given field ID. */ | ||
| ReadBuilder setCustomType(int fieldId, Class<? extends StructLike> structClass); | ||
|
|
||
| /** Set a filter to apply on result rows if applicable. */ |
There was a problem hiding this comment.
It is very important to mention that the caller still needs to do residual filtering, because some filters might not be supported, and some formats might not support filtering at all.
Something like this:
/**
* Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
* records. Some readers may not support filtering, or may only support filtering for certain expressions.
* In this case the reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
* apply the filter again.
*
* @param filter the filter to set
*/
| ReadBuilder setCustomType(int fieldId, Class<? extends StructLike> structClass); | ||
|
|
||
| /** Set a filter to apply on result rows if applicable. */ | ||
| default ReadBuilder filter(Expression newFilter) { |
There was a problem hiding this comment.
nit: maybe call the parameter filter
|
Took another look, and apparently our internal Parquet reader can only do row group filtering, so for InternalData that reads metadata (including partitions stats) probably such filtering won't bring any additional value, because such data rarely spans multiple row groups. |
2 participants
No description provided.