Support paritioned parquet dataset

[alippai]

File location patterns mydata/*.parquet and mydata/partition=01/*.parquet are common storage formats. Is it possible to support reading mydata in these cases? In theory Datafusion should have some support

[timvw]

I have asked and implemented the same globbing feature(s) in datafusion: isue-2393, but not sure how much of that is still supported after the recent refactorings...

In my example I have the following:

• s3://bucket/events/year=2022/month=4/day=5/part-xxx.c000.snappy.parquet
• s3://bucket/events/year=2022/month=4/day=4/part-xxx.c000.snappy.parquet
  The following just works: qv s3://bucket/events

Unfortunately, the following does not work (anymore):
qv "s3://bucket/events/year=2022/month=4/**/*.parquet"

[alippai]

Specifying the parent directory fails if other files are there eg. _SUCCESS generated by Apache Spark and mydata/*.parquet globbing doesn't work.

[timvw]

Correct (That reminds me of why I wanted the feature in the first place ;)).

Will look into this again once datafusion has onboarded the iox objectstore...

[alippai]

Makes sense, thank you.

[timvw]

In the newest release there is now support for globbing as well :) (
eg: qv "s3://datafusion-parquet-testing/data/alltypes_pla*n.parquet" works as expected

I have also updated the "listing" of files such that hidden files/directories such as_SUCCESS/_temporary are excluded