Default to collecting statistics when creating LIstingTables

[alamb]

Today, when creating tables of parquet files using CREATE EXTERNAL TABLE or ListingTables, statistics are not gathered.

This is good in the sense that creating the table is fast(er) but subsequent queries might be slower

The behavior is clarified in

• Clarify documentation about gathering statistics for parquet files #16157

@davisp suggests that defaulting to collecting statistics would make more sense (and I agree):

I’ll also note that my personal preference would be to default to true purely because it took a surprising amount of work to figure out how to even report #15908 not knowing that statistics collection was a config option. I do see the rationale around the behavior change, though I’d say either way that flag is defaulted is a behavior change and true seems like a saner default.

Originally posted by @davisp in #16080 (review)

[alamb]

I suggest we change the default value of datafusion.execution.collect_statistics so statistics are collected on table creation

[brayanjuls]

take

[davisp]

Registering my official +1 to default to collecting statistics.

For reference, I was working on the TPC-H benchmarks with a scale factor of 20 which generates roughly 20GiB of CSV data to process. Without statistics, query 17 or 18 would OOM on a 32GiB machine after about 20s. With statistics it never uses more than 1.9G and finishes in about 5s.

I generally agree that statistics collection is probably a bit slower than not collecting, but my guess is that any folks that notice and/or care would be doing something like high frequency queries against tiny datasets which seems like a niche use case. However, I’d wager a minute amount of money that the benefits would be worth the cost around the 500MiB to 1GiB range (based on my assumption that statistics collection is just scanning row groups, not full table scans).

[brayanjuls]

I am sorry for the delay on a resolution to this issue. I have been busy at work and the workload will remain the same at least for the next 3 weeks so I prefer to release this issue for someone else to take it and not become a blocker.

[alamb]

Thanks @brayanjuls -- hope all goes well at work

[AdamGS]

Is there anything to this issue besides changing the default datafusion.execution.collect_statistics, fixing any tests that rely on the default value being false and and changing docs accordingly? Would be glad to take it, just wondering if there's anything I'm missing.

[alamb]

Is there anything to this issue besides changing the default datafusion.execution.collect_statistics, fixing any tests that rely on the default value being false and and changing docs accordingly? Would be glad to take it, just wondering if there's anything I'm missing.

I think that just about covers the issue

[blaginin]

#16447 is merged so closing this