Fixes RemoveOrphanFiles delete files unexpected

[ConeyLiu]

RemoveOrphanFiles use actualFileDF leftanti join validFileDF to determine which files should be removed. We will list all the files under the provided or table location directory with FileSystem.listStatus and create the actualFileDF. validFileDF is created by index those metadata file and reference.

However, FileSystem.listStatus will qualify the given path. For example: a path: hdfs:/path will be qualified with hdfs://host:port/path. If the warehouse is set as: hdfs:/path:

validFileDF:
hdfs:/path/file1
hdfs:/path/file2
hdfs:/path/file3
....

actualFileDF:
hdfs://host:port/path/file1
hdfs://host:port/path/file2
hdfs://host:port/path/file3
....

Then, all the files in actualFileDF will be treated as invalid.

In this patch, we only compare the pure path (remove the schema and authority) when doing the leftanti join.

Updated existed UTs to test it.

[aokolnychyi]

There have been multiple discussions around this. I'll try to fetch the old thread on slack later today.

[ConeyLiu]

Hi @aokolnychyi, could you help to review this? Thanks a lot.

[ConeyLiu]

gentle ping @rdblue @aokolnychyi, could you help to review this? Thanks a lot.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[findepi]

Can this store data in memory?

[findepi]

In this patch, we only compare the pure path (remove the schema and authority) when doing the leftanti join.

This sounds reasonable to me, but there is one caveat.
The table files may span multiple buckets, so comparing without schema and authority may not be quite correct.
I think it's fine because

• in worst case we don't clean some orphan files, which isn't unsafe
• for tables spanning multiple locations, does orphan removal work at all?

but someone else should confirm that.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.