Support Reading PageIndex with ParquetRecordBatchStream

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

ParquetRecordBatchStream does not contain the necessary logic to read the offset index

Describe the solution you'd like

Add support for reading the column index to ParquetRecordBatchStream

Describe alternatives you've considered

Additional context

[thinkharderdev]

I can take this one

[tustvold]

I think this will involve reading the necessary data in ParquetRecordBatchStreamBuilder::new if the metadata returned by AsyncFileReader::get_metadata doesn't already have it loaded. I think it is important to handle the case where the AsyncFileReader is able to provide this information, e.g. if there is some metadata caching, the data is stored in a catalog, etc...

[thinkharderdev]

Was tinkering around with this and we can go one of two ways:

1. Fetch all the index metadata when constructing the stream.
2. Fetch the index on demand as needed.

The latter approach, we could add something to AsyncFileReader like

fn get_page_locations(&mut self, metadata: &ParquetMetadata, mask: &ProjectionMask) -> BoxFuture<'_, Result<Vec<Vec<PageLocation>>>

to only fetch what we need for a given projection. But that may be overkill given the amount of data....

[tustvold]

I think do the first option for now, the data should be small and relatively cheap to decode. We can always revisit if this simple approach proves insufficient