[Improvement]: Use partition filter to speed up optimizing plan

[HuangFru]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

AMS currently uses PartitionFilter to filter unnecessary files when planning the table before optimizing. But this only filters files and cannot speed up the plan because all the files have been planned. We should consider supporting partition filters on the scan.

How should we improve?

Support partition filter on scan to speed up plan.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[huyuanfeng2018]

I discussed this issue with @zhoujinsong before, let’s see if he has any opinions. The partition filter conditions cannot be passed in iceberg scan, but the manifest list contains the maximum and minimum values ​​​​of the saved partitions. can use this as an entry point to obtain it maybe.

[zhoujinsong]

Yes, @wangtaohz and @HuangFru shared a feasible transformation method to me today. I think this issue is related to that. I am looking forward to the implementation of this improvement.

BTW, this issue is related to #1842.

[wangtaohz]

Yes, @wangtaohz and @HuangFru shared a feasible transformation method to me today. I think this issue is related to that. I am looking forward to the implementation of this improvement.

We can transform partition data into a data filter expression, @HuangFru will submit a PR to support it.