[feature] Support failurePolicy like successPolicy

[gaocegege]

We should support custom failPolicy like successPolicy.

For example, when there is one master, 4 workers. When 4 workers are failed and 1 master is running, it is hard to say that the job is failed.

Thus we need to have a way to let users tell us when we should think the job is failed.

/cc @ChanYiLin

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/operator	0.64
feature	0.98

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[terrytangyuan]

Maybe failurePolicy to be more consistent?

[gaocegege]

SGTM

[ChanYiLin]

also SGTM

[pingsutw]

/assign

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[terrytangyuan]

/lifecycle freeze

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[terrytangyuan]

/lifecycle freeze

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.