Retry failed tasks

[marco-c]

Most of the failures are due to transient issues, we should implement some kind of retry mechanism.

Does it apply to static analysis too, @La0 @jankeromnes?

[mutterroland]

I would love to have some more infos. Maybe I can have a try on that one :).

[marco-c]

@mutterroland this issue requires knowledge of asyncio, are you familiar with it? If you aren't, I'd suggest working on another good-first-bug.

[mutterroland]

@marco-c I will move to another good-first-bug.

[jankeromnes]

Does it apply to static analysis too?

Yes, it just happened during today's release, because pulse-listener and static-analysis didn't update at exactly the same time (in this window, an incompatibility between bot versions caused a few analyses to fail).

[La0]

The retrigger code could be reused here.

[La0]

I'm starting to work on this one, here is what i can do:

1. Retry immediately tasks in exception state (with a maximum of N retries - N=3). Exception states are mostly due to Taskcluster (workers down, timeouts, ...)
2. Retry tasks in error state once, with a waiting time

If I use the retrigger code, a new task group & task id will be created, so I need to keep track of this in the code.
The rerunTask would be much cleaner but is marked deprecated (i asked on #taskcluster about an alternative).

[marco-c]

Sounds good to me, maybe the retry with a waiting time can be configurable per project (as it's possible different projects will have different needs).
I know tasks in error state for code coverage are often due to non-recoverable errors, it would be nice to have a way not to retrigger them, but it's not that easy to know whether an error is recoverable or not.

[La0]

From #taskcluster:

<&pmoore> | bastien: although it isn't documented there, i believe the preferred approach is to create a new task with the same task definition, but with a different taskId. we should probably update the docs
<&pmoore> | bastien: i've created taskcluster/taskcluster-queue#292 for this (feedback welcome as this is a particularly delicate issue - aki-away/bstack/dustin/jhford)

[La0]

Bumping this, as it's blocking static analysis users: mozilla/release-services#1846