[Improvement] Cleanup unused shuffle servers after stage completed

[xianjingfeng]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

If there are many stages in one applicaiton, spark driver will send heartbeat to every shuffle servers, and it may cause app expired in shuffle server side. And if we support decommission in the future, it will cause shuffle difficult to exit.

How should we improve?

Cleanup unused shuffle servers after stage completed

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[roryqi]

Spark has a mechanism called ExchangeReuse. It means that we can use the older stage' shuffle data.
We have already a pr #249, I don't know whether this pr can solve your problem.

[roryqi]

Have you joined our Wechat group? You can scan the QR code

[xianjingfeng]

Spark has a mechanism called ExchangeReuse. It means that we can use the older stage' shuffle data. We have already a pr #249, I don't know whether this pr can solve your problem.

#249 will clean the older stage' shuffle data, and it will not clean shuffle servers in shuffleServerInfoSet, right?

[xianjingfeng]

Have you joined our Wechat group?

No, how to join it?

[roryqi]

Spark has a mechanism called ExchangeReuse. It means that we can use the older stage' shuffle data. We have already a pr #249, I don't know whether this pr can solve your problem.

#249 will clean the older stage' shuffle data, and it will not clean shuffle servers in shuffleServerInfoSet, right?

Yes. you're right.

[roryqi]

Have you joined our Wechat group?

No, how to join it?

You can scan the QR code

[xianjingfeng]

Spark has a mechanism called ExchangeReuse. It means that we can use the older stage' shuffle data. We have already a pr #249, I don't know whether this pr can solve your problem.

#249 will clean the older stage' shuffle data, and it will not clean shuffle servers in shuffleServerInfoSet, right?

Yes. you're right.

So it is ok if we cleanup unused shuffle servers?

[roryqi]

Spark has a mechanism called ExchangeReuse. It means that we can use the older stage' shuffle data. We have already a pr #249, I don't know whether this pr can solve your problem.

#249 will clean the older stage' shuffle data, and it will not clean shuffle servers in shuffleServerInfoSet, right?

Yes. you're right.

So it is ok if we cleanup unused shuffle servers?

Yes, if we already removed the shuffle data in those shuffle servers, we can remove the shuffle servers in the shuffleServerInfoSet.