Operator should stop current load balancer job and start a new one if retry job constantly fails

[kevinliu24]

Introduction
Currently when a data balance job, the operator constantly retries a job until it succeed. Sometimes success is not possible (i.e. if the job was manually removed). We should detect the job status and retry only a few times after it fails. If it still fails, we should stop the current data balance job and start a new one.

Contents
If a load balance job fails after a few retries, stop the current job and start a new one.

Related work