[C++][Dataset][Python] ParquetFileFragment should provide access to parquet FileMetadata

[asfimport]

Related to ARROW-8062 (as there we will also need a way to expose the global FileMetadata). But independently, it would be useful to get access to the FileMetadata on each ParquetFileFragment (eg to get access to the statistics).

This would be relatively simple to code on the Python/R side, since we have access to the file path, and could read the metadata from the file backing the fragment, and return this as a FileMetadata object.

I am wondering if we want to integrate this with ARROW-8062, since when the fragments were created from a _metadata file, a ParquetFileFragment.metadata attribute would not need to read it from the parquet file in this case, but from the global metadata (at least for eg the row group data).

Another question: what for a ParquetFileFragment that maps to a single row group?

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Ben Kietzman / @bkietz

Related issues:

• [C++][Dataset] Parquet Dataset factory from a _metadata/_common_metadata file (is related to)

PRs and other links:

• GitHub Pull Request #7523
• GitHub Pull Request #7546

_{Note: This issue was originally created as ARROW-8733. Please see the migration documentation for further details.}

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
We could expose this yes. In ARROW-8062, I'm constructing the ParquetFileFragment without holding the original FileMetaData for mutltiple reasons, but more importantly because it may or not map to what the real physical FileMetaData holds. I think that it would be ill conceived to create a fake FileMetaData constructed from the _metadata. I plan to expose the statistics probably via shared_ptr.

There is various details to flesh out regarding this, especially the one where a fragment only contains a strict subset of the row groups.

What about serialization for dask? That's another potential issue.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

but more importantly because it may or not map to what the real physical FileMetaData holds

Do you mean because the _metadata file might not be correct (I think we can assume it is, responsibility of the user to ensure the integrity f the dataset?), or rather that the metadata might not map anymore to the exact fragment for other reasons? (eg we might already have filtered out some fragment / parts of fragments)

The file paths contained in the metadata of a _metadata file will of course also not map to the one in the physical metadata of the actual data file.

I plan to expose the statistics probably via shared_ptr.

The idea here would then be to expose statistics of a Fragment in a file format-agnostic way?

What about serialization for dask? That's another potential issue.

The FileMetadata itself are already serializable in Python, as well as expressions.

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
I mean that the _metadata's FileMetaData might differ significantly from the file's FileMetaData on disk. For example, num_rows will be out of sync, or the encryption might be different per-file.

So, do we re-use the FileMetaData taken from the _metadata file (if constructed that way), or do we always pay the filesystem cost whenever the user requests this metadata?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Hmm, good questions. Currently in ParquetDataset, we expose this metadata, and dask actually uses it (but only for the statistics, not for something like the encryption information).

cc @rjzamora

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

So, do we re-use the FileMetaData taken from the _metadata file (if constructed that way), or do we always pay the filesystem cost whenever the user requests this metadata?

For dask's use case, being able to access the metadata (or at least the statistics) for a certain fragment without the filesystem cost (in case the dataset/fragments were constructed with the ParquetFactory from a _metadata file) is quite important I think.

And I think ideally this can be accessed from the Fragments. Because if the fragments are filtered, there is no easy way for dask to process the _metadata file themselves to gather all the statistics, as they would then need to repeat the filtering to get the statistics matching the filtered fragments .. (which is not going to very robust if there are small variations in filtering logic, plus it would be duplicating this logic).

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

I mean that the _metadata's FileMetaData might differ significantly from the file's FileMetaData on disk. For example, num_rows will be out of sync, or the encryption might be different per-file.

Indeed those things might be different or not relevant for the fragment.
However, I think that for dask, they only need the statistics. Would it therefore be an option to only expose this? Or only the RowGroupMetadata, which should be file specific?

[asfimport]

Rick Zamora / @rjzamora:
Although the current dask code is using pyarrow to construct a list of statistics for all row-groups in the dataset (ideally using __metadata), I am certainly willing to put in the effort to relax this approach a bit. If pyarrow is capable of returning a list of filtered row-group fragments, the only other thing dask really needs is the min/max value of the index column for each of those fragments (assuming the index must be specified by the user or the "pandas" metadata).  This is all to say that Dask doesn't need to inspect the _metadataitself if the relevent statistics for the actual fragments are already available.

[asfimport]

Ben Kietzman / @bkietz:
Issue resolved by pull request 7546
#7546