[Datasets] Cannot Read parquet from multiple directories in aws s3.

[fodrh1201]

Description

Error occurs when reading parquet files from multiple directories in aws s3.

• Read multiple directories.

ray.data.read_parquet(["s3://bucket/path1", "s3://bucket/path2"])

OSError: Error creating dataset. Could not read schema from 's3://bucket/path1': Path does not exist 's3://bucket/path1'. Is this a 'parquet' file?

Use case

I think it would be nice if you could read it in multiple directories like read_json or read_csv.

[jianoaix]

If you are blocked, you may try this to achieve the same:
ds1 = ray.data.read_parquet(["s3://bucket/path1"])
ds2 = ray.data.read_parquet(["s3://bucket/path2"])
ds = ds1.union(ds2)

[Joeavaikath]

This is still an active issue, documentation states multiple directories are allowed. Is that only for local and not remote?

[clarkzinzow]

@Joeavaikath This unfortunately doesn't work with ray.data.read_parquet() since it uses Arrow's Dataset abstraction under-the-hood, which doesn't accept multiple directories as a source: https://arrow.apache.org/docs/python/generated/pyarrow.dataset.dataset.html

In addition to the workaround that @jianoaix mentioned, we also have a ray.data.read_parquet_bulk() API which does not use Arrow's Dataset abstraction and would work with multiple directories if the right metadata provider is given:

import ray

ds = ray.data.read_parquet_bulk(
    ["s3://ursa-labs-taxi-data/2019/01", "s3://ursa-labs-taxi-data/2019/02"], 
    meta_provider=ray.data.datasource.DefaultFileMetadataProvider(),
)

Note that this Parquet reading API will not inline Hive directory-partitioned key-value pairs into the table, so your partition columns must exist both in the path and in the Parquet files.

I'll open up a PR that fixes the read_parquet() docs to make this clear that multiple directories are not allowed.