cylc message should have submit number info

[matthewrmshin]

Long story short, something bad happened to a task host, and we ended up with something like this:

1. Suite got a message from (submit 1 of) a task when it died with a TERM signal.
2. Task host became unavailable for a time.
3. Task host came back.
4. Suite (retry) submitted job 2 of the task to the task host.
5. Suite got a message from (submit 2 of) the task that it succeeded.
6. For some reason, job 1 was resumed on the task host, which then sent a message back to the suite to say that it failed.
7. Tasks downstream of this task were a bit messed up.

If we have the submit number in the task message, the suite should be able to discard messages from earlier submits.

[matthewrmshin]

May be worth looking at #439, #2214 and/or #2502 when we look at this one.

[matthewrmshin]

The other usual way to trigger this problem is when users reset the state of a submitted or running task to ready - without killing the original job first. This should also be handled correctly. Should we have the suite make an automatic attempt to kill the original job when user resets the state of a submitted or running task?

[hjoliver]

Should we have the suite make an automatic attempt to kill the original job when user resets the state of a submitted or running task?

I think yes. It doesn't really make sense to have to two instances of the same task job running at once.

[hjoliver]

Somewhat relatedly (although not a submit number issue): careless (but common) use of suicide triggers can result in suiciding an active task proxy. Currently we just log a warning about this; we should probably kill the active job as well.

[matthewrmshin]

#2505 not really related, but I'll put a link here any way.