TES backend?

[buchanae]

I'm exploring the possibility of cwltool being able to submit tasks to a TES server as a means of cluster orchestration.

As I understand it, this has been sort of tried before in funnel-client. There's also possibly something similar in arvados-cwl. I'm not sure about arvados_cwl, but I've heard it was difficult to keep funnel-client working because the integration with cwltool was tricky.

Mostly I'd like to get the opinion of the devs here who are familiar with the cwltool code. Is there a clean way to write a new executor plugin? Is this something you'd be interested in having in the cwltool codebase?

[tetron]

Hi @buchanae

Yes, this is something we'd like to have.

The way Arvados integrates/extends cwltool has been to add hooks to override functionality on an as-needed basis but I think we could formalize it so there is a documented way to add backends. I took a stab at documenting the current extension points here:
https://github.com/common-workflow-language/cwltool#extension-points

I think the ideal way to do this would be something like:

1. Add a plugin system, eg support plugins for third party CWL registry APIs, etc #314
2. Refactor so that plugins can set the extension points listed above
3. Implement a TES plugin using those extensions

I don't have bandwidth to work on this right now, but I'm happy to support anyone who wants work on it.

[buchanae]

Thanks for the info, and glad to hear you're interested.

Does #314 block working on extension points (item 2)? I'm not sure I understand #314, but it looks specific to resolving tools?

[tetron]

The extension points mostly already exist, but without a plugin scheme you have to create a new application which imports cwltool in order to use them (which is how arvados-cwl-runner currently works, and funnel-client).

A plugin scheme would provide necessary motivation refactor/formalize/document the extension points.