[Improvement]: Make the maximum input file size for per optimize thread configurable

[hameizi]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

Currently, the maximum input file size for each optimize thread is fixed at 512MB. When processing tables with rapidly generated data, having a single optimize thread handle fewer files can cause the optimization merging speed to fall behind the rate at which the table is generating files. In cases where concurrent resources for optimization are sufficient, increasing the number of files processed by a single optimize thread can effectively address this issue.

How should we improve?

Make the maximum input file size for per optimize thread configurable

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• I agree to follow this project's Code of Conduct