Parquet ArrayReader should allow reading a subset of row groups

[alamb]

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-11016

Parquet ArrayReader currently only supports reading an entire file from start to finish and does not allow selectively reading a subset of row groups. This prevents us from parallelizing work across threads when processing a single parquet file.

[alamb]

Comment from Andy Grove(andygrove) @ 2020-12-23T16:28:31.604+0000:

[~nevi_me] [~sunchao] Do either of you know if this would be a lot of work to implement or not? If you have any pointers for someone working on this ticket it would be appreciated.

Comment from Neville Dipale(nevi_me) @ 2020-12-24T19:37:44.895+0000:

I'm not yet sure about what it would take to read the same file across threads. Would that be like sharing the file handle, as that might not work on all OSes.

 

From working with the arrow -> parquet IO, I think that if the batch size is aligned with record group size, it might make it easier to read partial data from 1 or more groups.

I'll have a look in the coming days, and update this ticket with more info. It might likely be a case of exposing some method to do partial reads.

Comment from Chao Sun(csun) @ 2020-12-29T01:44:48.654+0000:

Sorry for the late reply. Yes I think it should be possible. On the file reader side we can pass in a (start, end) besides the file handle, to indicate we want to only read a segment of the file. Then after parsing the file metadata, we can check all the row groups for the file and determine which row group(s) overlaps with the segment, and only select those. 

You can probably check relevant code in [Spark|https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java#L105] and [Parquet|https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L1223] for reference.

I'm not sure about the file handle sharing issue [~nevi_me] mentioned tho - I thought we used to clone file handle so that they can be shared but yeah haven't looked at the code base for some time :(