Feature sbg task retries

[yfan2012]

This PR adds task retry functionality to the SBG platform as the launcher waits for tasks to complete. This is implemented in the monitor_task() function which does the following:

• for SBG: the function checks the task status every 60 seconds. If it finds that a task exited with a pre-specified list of error messages (defined inSevenBridgesInstance class), it will resubmit the task up to three times on progressively larger instances also defined in the SevenBridgesInstance class. It returns either a completed task or a failed task that has been retried.
• for arvados: the function checks the state of the task every 60 seconds. It exits and returns either a completed or failed task.

[golharam]

We already have a function call get_task_state(). This PR proposes a higher level method called monitor_task() that will return when the task succeeds or fails.

[golharam]

Why are there instance types hard-coded here? This assumes that is what should be used. There doesn't seem to be a way to override that either.

[ryanporterfield]

Hello @golharam and @yfan2012,

I see what you're saying here. When I wrote this, I just had the 10X launcher in mind. I forgot that the sevenbridges_platform.py should work for all the launchers. I can implement something similar to how the launcher functioned previously and move the instance types to the config file. I can also add arguments to specify whether or not a task should be retried, as well as a specific instance type to re-run the task on.

Is this something you'd like us to work on?

[golharam]

Yes. I want to make sure this package stays generic enough to not include anything specific to an individual pipeline. The config settings should be part of the launcher for the pipeline.

[ryanporterfield]

Hey @golharam,

I rewrote this logic to be launcher agnostic, but need permissions to create a PR on this repo. Would you mind granting me those? And the rest of our team if that's okay with you?

Ryan