Remove get_scan_files and ExecutionPlan::file_scan_config (#7357)

[tustvold]

Which issue does this PR close?

Closes #7357

Rationale for this change

This was added by @yahoNanJing in #5572. It was then evolved in #7175 to try to avoid strongly coupling to AvroExec. However, this is causing issues trying to split apart the crates (#7357) and it is unclear how best to adapt the design #7425 (comment).

Given this API does not appear to be being used, I wonder if we can do the simplest possible thing, and just remove it

What changes are included in this PR?

Removes the API

Are these changes tested?

Are there any user-facing changes?

[crepererum]

Looks good.

Let's get @not-my-profile's view on that, i.e. if he has a good reason to keep this API.

[not-my-profile]

get_scan_files is public API. So just because it's not used internally doesn't mean that it's not used at all.

The question is if @yahoNanJing's motivation still applies:

Currently the TreeNodeRewriter is as a visitor to transform a node to another. However, sometimes we don't need to do the transformation and what we want is only to collect some info from the node. To achieve this, it's better to introduce another visitor for collecting info and keep the node unchanged.

As I said in #7425 (comment) I'm not familiar enough with datafusion to determine that.

it is unclear how best to adapt the design

I think that #7485 is a decent solution if we decide to keep the function.

It is used in ballista

[alamb]

It appears that get_scan_files is used in ballista: https://github.com/search?q=repo%3Aapache%2Farrow-ballista%20get_scan_files&type=code

[tustvold]

It appears to only be used as an argument for https://github.com/apache/arrow-ballista/blob/948d0777f972144cc242d0398fd61fadf34cec73/ballista/scheduler/src/cluster/mod.rs#L679 which seems like it could definitely be achieved in a different way

[alamb]

I looked more into the ballista code and I think the files are used for more than just their file count. Specifically the URL is used here:

https://github.com/apache/arrow-ballista/blob/948d0777f972144cc242d0398fd61fadf34cec73/ballista/scheduler/src/cluster/mod.rs#L625C1-L626C1

I am thinking, however, that we could effectively port get_files_for_scan into Ballista... Let me give that a try

[alamb]

I moved this API into Ballista apache/datafusion-ballista#877 so I think it is ok to remove from DataFusion. @yahoNanJing please advise if there are reasons to avoid this