Enable parquet filter pushdown by default

[alamb]

In #3380 @thinkharderdev added support for evaluating filters during the parquet scan via the RowIndex mechanism 🎉

This feature is currently enabled via a feature flag, which is disabled by default.

This ticket tracks enabling this feature by default.

Currently known items are:

• Add benchmarks for parquet queries with filter pushdown enabled #3457
• Review of parquet test coverage and add additional coverage as needed: Correctness integration test for parquet filter pushdown #3976
• Incorrect results with parquet filtering pushdown enabled #4005
• Internal error when parquet predicate pushdown is enabled "Error evaluating filter predicate:" #4006
• Another Internal error when parquet predicate pushdown is enabled "Error evaluating filter predicate: #4046
• Enable this feature by default: Enable Parquet Row and Page Filtering by default (WIP) #3828

[alamb]

I plan to review the parquet test coverage over the next day or two

My basic plan is to:

1. Make a draft PR that enables pushdown by default and see if any regression tests fail
2. Do some manual testing using datafusion-cli and some internal datasets

[alamb]

The PR that enables the feature is #3828 and it looks quite promising

Also, I have a proposed config change that I would like to get in, #3822, which would allow users to quickly disable this feature if they found issues

[alamb]

Update here: I am working on some fuzz testing for the predicate pushdown code

[alamb]

I found a bug in pushdown by writing a test #4005

[alamb]

I found another one with my test. I will keep the list on this ticket updated

[alamb]

I think we are getting close

[alamb]

Update here is I think once we get #3976 merged I'll put up the PR to enable the feature by default

[Dandandan]

@alamb What's the remaining work that needs to be done to enable it by default? Anything I can do to get this over the finish line?

[alamb]

@Dandandan ❤️ thank you

I know of two major items:

1. Enable page index by default Enable parquet page level skipping (page index pruning) by default #4085
2. Ensure that enabling filter pushdown by default doesn't cause performance regressions in cases where the filter isn't very selective. I know @tustvold had some thoughts in this area.

For item 1, I tried a few times, but am currently blocked by #5104 and haven't had time to return to it

For item 2, I think someone needs to look at some benchmark results (perhaps TPCH) and figure out what we can do to avoid the regression (or measure and determine it isn't significant)

I keep hoping to have time to work on item 1, as I am pretty sure I know what the problem is, but other things keep coming up :(

I haven't had a chance to work on 2 yet

[alamb]

Update here is I am working on a larger benchmarking story, part of which would give us more confidence to merge changes like this in. I hope to have that done early next week

[alamb]

With the introduction of bench.sh with tpch and clickbench (#6994) I think we are now in a better position to do this experiment again and fix any performance regressions

[tustvold]

apache/arrow-rs#5523 might help mitigate the impact of pushing down predicates that turn out to not be very selective

[alamb]

It would be really nice to (finally) be able to turn this optimization on -- thank you @tustvold

cc @Dandandan

[alamb]

I tried again now that #4028 is done in #12524. Sadly many of the queries still slow down. I will try and find time to figure out why

[Dandandan]

@alamb I wonder if you already have some further insights?

[Dandandan]

I am just reading the latest messages on apache/arrow-rs#5523 , thanks for the updates

[alamb]

The writeup here from @tustvold is quite good: apache/arrow-rs#6454 (comment)