[Improvement] Implement the interface ShuffleDataIO

[roryqi]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

We can implement ShuffleDataIO
https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/shuffle/api/ShuffleDataIO.java
https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/shuffle/api/ShuffleDriverComponents.java
Dynamic Allocation don't need patch any more.

How should we improve?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[beryllw]

I'm Interested in this, could you assign it to me？

[roryqi]

I'm Interested in this, could you assign it to me？

OK, I have assigned to you.

[beryllw]

The patch [SPARK-42689] that allows ShuffleDriverComponent to declare if shuffle data is reliably stored has been fixed in Spark 3.5, which has not yet been released. Do I need to test it?

[roryqi]

Yes, you can build a spark client with master branch to test it.

[beryllw]

Yes, you can build a spark client with master branch to test it.

Okay, I will do that.

[summaryzb]

volunteering

[roryqi]

@Kwafoor If you're busy recently, let @summaryzb finish this issue first.