[Python][Dataset] Detect and use _metadata file in a list of file paths

[asfimport]

From dask/dask#6047 (comment)

When specifying a directory to ParquetDataset, we will detect if a _metadata file is present in the directory and use that to populate the metadata attribute (and not include this file in the list of "pieces", since it does not include any data).

However, when passing a list of files to ParquetDataset, with one being "_metadata", the metadata attribute is not populated, and the "_metadata" path is included as one of the ParquetDatasetPiece objects instead (which leads to an ArrowIOError during the read of that piece).

We could detect it in a list of paths as well.

Note, I mentioned ParquetDataset, but if working on this, we should probably directly do it in the datasets API-based version.
Also, I labeled this as Python and not C++ for now, as this might be something that can be handled on the Python side (once the C++ side knows how to process this kind of metadata -> ARROW-8062)

Reporter: Joris Van den Bossche / @jorisvandenbossche

Related issues:

• [Python][C++][Dataset] Implement split_row_groups for ParquetDataset (relates to)
• [Python][C++] Possibly use _common_metadata for schema if _metadata isn't available (is related to)
• [Python][C++] Document how to write _metadata, _common_metadata files with Parquet datasets (is related to)
• [C++][Dataset] Parquet Dataset factory from a _metadata/_common_metadata file (is related to)

_{Note: This issue was originally created as ARROW-8446. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
I added links to a number of interrelated issues