[SPARK-3831] [SQL] Filter rule Improvement and bool expression optimization.

[sarutak]

If we write the filter which is always FALSE like

SELECT * from person WHERE FALSE;

200 tasks will run. I think, 1 task is enough.

And current optimizer cannot optimize the case NOT is duplicated like

SELECT * from person WHERE NOT ( NOT (age > 30));

The filter rule above should be simplified

[SparkQA]

QA tests have started for PR 2692 at commit 8ea872b.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2692 at commit 8ea872b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21370/Test FAILed.

[yhuai]

@sarutak LGTM. Can you take a look at the failing test?
The log is

[info] - NOT (i < 88) *** FAILED ***
[info]   2 did not equal 10 Wrong number of read batches (PartitionBatchPruningSuite.scala:91)

Seems we need to update the test suite since with your change, we can handle this predicate when doing batch pruning for cached tables. Also, it will be good to add another case involving NOT into "unsupported predicate" if possible.

[sarutak]

@yhuai Thanks picking this PR up and for your comment!
I'll try soon.

[SparkQA]

QA tests have started for PR 2692 at commit a11b9f3.

• This patch merges cleanly.

[yhuai]

How about

checkBatchPruning(s"NOT (i in (${(1 to 30).mkString(",")}))", 31 to 100, 5, 10)

For this case, we will read 4 partitions including 7 batches when we can support it.

[SparkQA]

QA tests have finished for PR 2692 at commit a11b9f3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21372/Test PASSed.

[sarutak]

@yhuai Thanks, it makes sense.

[SparkQA]

QA tests have started for PR 2692 at commit 23c750c.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2692 at commit 23c750c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21381/Test PASSed.

[yhuai]

LGTM

cc @marmbrus.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21470/Test FAILed.

[sarutak]

retest this please.

[SparkQA]

QA tests have started for PR 2692 at commit 25f3e20.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2692 at commit 25f3e20.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21473/Test PASSed.

[marmbrus]

Thanks! I've merged to master.