[Improvement] Optimize retry logic in `ShuffleServerGrpcClient#sendShuffleData`

[xianjingfeng]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

Now in org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient#sendShuffleData, it will retry to send to one shuffle server for a long time and fail after reach rss.client.send.check.timeout.ms. Exception as follows:

Timeout: Task[2852_0] failed because 200 blocks can't be sent to shuffle server in 600000 ms.

This will cause that client will not send data to other servers.

How should we improve?

1. Don't retry in requirePreAllocation and just retry in upper level
2. Set the default value of rss.client.retry.max to a smaller value, such as 10.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[zuston]

Maybe this has been optimized in #332 . cc @jerqi

[xianjingfeng]

It seems not. This issue describes that when there are multiple replicas, if the first server can't be written, we should switch to other servers as soon as possible.

[zuston]

Make sense.

But this may be hard to control. If you have some ideas, please go ahead

[jerqi]

I don't suggest that we modify the default value. Because we should verify the parameters in our production environment with huge amount of data first.

[xianjingfeng]

I don't suggest that we modify the default value. Because we should verify the parameters in our production environment with huge amount of data first.

But the current default value is obviously unreasonable.
rss.client.send.check.timeout.ms=600000 rss.client.retry.max=100 rss.client.retry.interval.max=10000
It will not send data to other servers if the first server blocked.
And after #159, it will retry 300 times

[xianjingfeng]

Make sense.

But this may be hard to control. If you have some ideas, please go ahead

I just change the default value and adjust the retry logic introduced by #159

[jerqi]

I don't suggest that we modify the default value. Because we should verify the parameters in our production environment with huge amount of data first.

But the current default value is obviously unreasonable. rss.client.send.check.timeout.ms=600000 rss.client.retry.max=100 rss.client.retry.interval.max=10000 It will not send data to other servers if the first server blocked. And after #159, it will retry 300 times

Is it ok for single replica?

[xianjingfeng]

Is it ok for single replica?

Yes. But I think we usually use multiple replicas.If the user is not familiar with this mechanism, he will think that fault tolerance can be achieved without modifying other configurations.

[xianjingfeng]

Do you have any better ideas? @jerqi

[jerqi]

Do you want to solve to avoid sending too much data to a heavy load server?

[xianjingfeng]

Do you want to solve to avoid sending too much data to a heavy load server?

No, i just want to solve that when the first shuffle server can't be written, we can send data to another one.

[jerqi]

How can we judge whether the server is writable?

[xianjingfeng]

How can we judge whether the server is writable?

We can set a timeout to per server. The current problem is that there is only the timeout for block sending and the time to send data to one server is bigger than the timeout for block sending

[jerqi]

How can we judge whether the server is writable?

We can set a timeout to per server. The current problem is that there is only the timeout for block sending and the time to send data to one server is bigger than the timeout for block sending

Could we add a config option for multi replicas timeout? If we use multi replicas, we could use this timeout.

[xianjingfeng]

Could we add a config option for multi replicas timeout? If we use multi replicas, we could use this timeout.

It's a little complicated. How about set real timeout to rss.client.send.check.timeout.ms * 2 if rss.data.replica.skip.enabled=true

Another problems:

1. Before [ISSUE-76] Disallow sendShuffleData if requireBufferId expired #159 it will retry 100 times, but it will retry 300 times after it. Should we set rss.client.retry.max=33 for recovery it?

rss.client.send.check.timeout.ms=600000 rss.client.retry.max=100 rss.client.retry.interval.max=10000

100 * 10000 = 1000000 > 600000, This default values seems unreasonable.