HadoopFileIO to support bulk delete through the Hadoop Filesystem APIs

[steveloughran]

Feature Request / Improvement

Hadoop Filesystems now support paged bulk delete API.

For most filesystems the page size is 1; it simply mapped to a single file delete.

For S3A, the page size is the value of fs.s3a.bulk.delete.page.size
-each page of deletions is executed as a single bulk delete POST in the AWS API

There are no attempts to implement POSIX "safety checks" for the path being a directory,
the parent directory existing afterwards etc.

As such it is the most efficient way to delete many objects, whose performance
should match that of to S3FileIO.deleteFiles().
For filesystems without bulk delete, this is mapped to delete(path), so
no less efficient than normal delete calls.

Support this in Iceberg:

• Add new option iceberg.hadoop.bulk.delete.enabled (default: false)
• Use reflection to use the bulk delete API through the reflection friendly
  org.apache.hadoop.io.wrappedio.WrappedIO class
• Switch to using the bulk delete mechanism if enabled and present

If active, bulk delete is supported by:

1. building up a page of paths to delete for every target filesystem.
2. Initiate an asynchronous bulk delete request whenever a page is full.
3. When the end of the list has been reached: queue page deletes for
   all incomplete pages.
4. Await results and report failures as such.

Missing files are not reported as failures -these are not detected.
Failures will be in permissions, network and possibly transient endpoint issues.

Adds a parameterized test to verify bulk delete works.
This needs to be run against hadoop 3.4.1 to actually verify coverage.

Testing this feature all the way to S3 is complicated.
A test within the hadoop-aws module can validate
the feature through HadoopFileIO and act as regression
testing for the S3A Connector.

Query engine

None

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'