select count(*) from '*.parquet' scans all parquet file recursively including subdirectory

[zhangxffff]

Describe the bug

I find select count(*) from '*.parquet' not only scan the parquet file in current directory, but it also recursively scan all the parquet file in subdirectory. I wonder is this behavior by design or a bug.

To Reproduce

I tried with three parquet file, and two of them are in subdir.

$$ tree
.
├── subdir
│   ├── file1.parquet
│   └── file2.parquet
└── users.parquet

2 directories, 3 files

users.parquet has 2 record, file1.parquet has 1 record and file2.parquet has 1 record.
select count(*) from '*.parquet' get 4

$$ datafusion-cli -c "select count(*) from '*.parquet'"
DataFusion CLI v33.0.0
+----------+
| COUNT(*) |
+----------+
| 4        |
+----------+
1 row in set. Query took 0.058 seconds.

$$ datafusion-cli -c "select count(*) from 'users.parquet'"
DataFusion CLI v33.0.0
+----------+
| COUNT(*) |
+----------+
| 2        |
+----------+
1 row in set. Query took 0.002 seconds.

$$ datafusion-cli -c "select count(*) from 'subdir/*.parquet'"
DataFusion CLI v33.0.0
+----------+
| COUNT(*) |
+----------+
| 2        |
+----------+
1 row in set. Query took 0.002 seconds.

Expected behavior

I try same query in duckdb, and duckdb only scan parquet file in current directory

$$ duckdb -c "select count(*) from '*.parquet'"
┌──────────────┐
│ count_star() │
│    int64     │
├──────────────┤
│            2 │
└──────────────┘

Additional context

No response

[alamb]

I think the reason this happens is that the ListingTable does ObjectStore::list which finds files in all subdirectories

I wonder is this behavior by design or a bug.

As I understand it, DataFusion is trying to model the behavior of "Hive PartitionedTables" -- so to answer this question I think we need to research what Hive does in this case

[zhangxffff]

I think the reason this happens is that the ListingTable does ObjectStore::list which finds files in all subdirectories

I wonder is this behavior by design or a bug.

As I understand it, DataFusion is trying to model the behavior of "Hive PartitionedTables" -- so to answer this question I think we need to research what Hive does in this case

I tried with hive external table stores as parquet, it seems that hive external table also do not scan parquet file in subdirectory

as show in this picture, when location is hdfs:///user/hive/warehouse/zxf_test/, there is no data in external table, when localtion is hdfs:///user/hive/warehouse/zxf_test/subdir, external table has two records from two parquet file.

I also tried partitioned external table.

After create table, there is no data.

After specify location of partition pt1, we can get data from hdfs:///user/hive/warehouse/zxf_test_pt/pt1

After also specfy location of partition pt2, we can get data from both hdfs:///user/hive/warehouse/zxf_test_pt/pt1 and hdfs:///user/hive/warehouse/zxf_test_pt/pt2

If I copy a subdirectory with parquet file into hive partition directory, hive report a java.io.IOException:java.io.IOException: Not a file

So it seems that hive also do not scan parquet file in the subdirectoy. for hive partitioned table, user should specify the directory of each partition, and there should not contains any subdirectory.

[alamb]

Makes sense to me. THank you for the research @zhangxffff

If we wish to change this behavior, perhaps we can add a configuration parameter to have the old or new behavior (defaulting to the new behavior)