[Python][C++] Possibly use `_common_metadata` for schema if `_metadata` isn't available

[asfimport]

Currently pyarrow's parquet writer only writes _common_metadata and not _metadata. From what I understand these are intended to contain the dataset schema but not any row group information.

 

A few (possibly naive) questions:

 

1. In the __init__ for ParquetDataset, the following lines exist:

if self.metadata_path is not None:
    with self.fs.open(self.metadata_path) as f:
        self.common_metadata = ParquetFile(f).metadata
else:
    self.common_metadata = None

I believe this should use common_metadata_path instead of metadata_path, as the latter is never written by pyarrow, and is given by the _metadata file instead of _common_metadata (as seemingly intended?).

 

2. In validate_schemas I believe an option should exist for using the schema from _common_metadata instead of _metadata, as pyarrow currently only writes the former, and as far as I can tell _common_metadata does include all the schema information needed.

 

Perhaps the logic in validate_schemas could be ported over to:

 

if self.schema is not None:
    pass  # schema explicitly provided
elif self.metadata is not None:
    self.schema = self.metadata.schema
elif self.common_metadata is not None:
    self.schema = self.common_metadata.schema
else:
    self.schema = self.pieces[0].get_metadata(open_file).schema

If these changes are valid, I'd be happy to submit a PR. It's not 100% clear to me the difference between _common_metadata and _metadata, but I believe the schema in both should be the same. Figured I'd open this for discussion.

Reporter: Jim Crist / @jcrist

Related issues:

• [Python][Dataset] Detect and use _metadata file in a list of file paths (relates to)
• [Python] Partition columns are not correctly loaded in schema of ParquetDataset (is related to)
• [C++][Dataset] Parquet Dataset factory from a _metadata/_common_metadata file (is related to)

_{Note: This issue was originally created as ARROW-2079. Please see the migration documentation for further details.}

[asfimport]

Jim Crist / @jcrist:
cc @xhochy

[asfimport]

Uwe Korn / @xhochy:

1. Yes this should be common_metadata_path but we could also use here metadata_path is common_metadata_path is None.

2. yes, this also looks fine.

 

_common_metadata is simply a Parquet file that contains the schema of all files that are part of a dataset. _metadata is the same but it additionally contains the metadata of each RowGroup that is part of the dataset, i.e. _metadata is a superset from _common_metadata.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
It seems to me that this is closed by ARROW-2209 (#1656).

At least, the two items mentioned above have been done by that PR (fixing the metadata_path -> common_metadata_path mentioned in 1), and validate_schemas now also falls back to common_metadata_path to get the schema).

—

One aspect I don't fully understand is the "Currently pyarrow's parquet writer only writes _common_metadata and not _metadata", as it seems to me that pyarrow does not write any of both? (at least not with write_to_dataset)

[asfimport]

Wes McKinney / @wesm:
We read it but don't write it. We might want to change that in the course of addressing ARROW-1983

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
But the _common_metadata we could already write now? As far as I understand, _common_metadata contains the metadata/schema of the full dataset, without the additional combined metadata for each dataset piece (which is what ARROW-1983 tries to address)

[asfimport]

Wes McKinney / @wesm:
Sure, we certainly could. I don't think it would make sense to do this independently from addressing the _metadata file though

[asfimport]

Uwe Korn / @xhochy:
It does make sense to write _common_metadata without _metadata as this already gives you a global schema for all Parquet files in the dataset. In the presence of _metadata, _common_metadata is actually redundant as the schema is already contained in _metadata.

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
Question to users/developers, why the need of 2 files, is it because _metadata can be too big?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
cc @rjzamora @martindurant

[asfimport]

Martin Durant / @martindurant:

 why the need of 2 files

 

Exactly - the _metadata file also contains all the column stats and encodings/offsets for every row-group in every file; __common__metadata contains only the schema and key/value global metadata.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
In practice, what does dask do? For _metadata, there is the option to disable it with write_metadata_file=False.
But will _common_metadata always be written?

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
@martindurant, my question is more about "why" than "how".

[asfimport]

Martin Durant / @martindurant:
to_parquet currently has write_metadata_file=True by default.

I believe that if it is False, you get neither file - this could be easily changed, however.

 

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@martindurant so in practice (when using dask), you will always have either both or either none, I understand?
On the reading side, has dask an API to indicate to use _common_metadata but not _metadata ?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
And the reason we are asking those questions is just to understand the use cases of _common_metadata, as the question is if it is worth supporting this in Arrow (we now support creating a dataset from _metadata, ARROW-8062)