[DSIP-69] Fix master dispatch task timeout might cause task duplicate running in worker

[ruanwenjun]

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Motivation

Right now, there exist some case might cause the task duplicated dispatched.
e.g.

The master dispatch task a to worker A first, but receive a timeout response, this might happen when the worker rpc is busy, then master will select a new worker B and retry the dispatch.

Then there might exist two situations:

1. The task has been received by worker A, then take will duplicate exist in worker A and worker B, both the two worker will execute the task, a worser case is the task might duplicated in more worker.
2. The task hasn't been received by worker A, then task will not duplicate executed.

The first situation is not accepted.

Design Detail

In order to solve this, we should change the dispatch logic.

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

No response

Code of Conduct

• I agree to follow this project's Code of Conduct