job host set-up by batch scheduler

[hjoliver]

PBS apparently has the ability to create directories and install files on the job host, before executing a job, and to push files back after the job.

On systems with this capability Cylc could presumably use it to avoid any need for non-interactive ssh to job hosts, which would make certain security-minded HPC people very happy.

So .. maybe we could move job host set-up into our "batch scheduler" abstraction (defaulting to current use of ssh as required).

[matthewrmshin]

(Sounds good, but we need to understand how this mechanism actually works under PBS.)

[hjoliver]

According to (a quick glance at) the PBS Pro User Guide http://www.pbsworks.com/pdfs/PBSUserGuide13.0.pdf (see chapter 4) - files and directories can be "staged in" before a job and "staged out" afterward, either to PBS-controlled secret run directory (which we don't want) or with an absolute path (which we do want) via -W stagein=... and -W stageout=... directives. The directive args specify a host and location for the stagein source, and for the stageout target, as well as the location on the execution host.

Also, from talking to PBS people, the method by which the staging is done is configurable e.g. to cp (if possible) or scp (which would presumably still require non-interactive ssh - but if initiated from the job host, which isn't clear, that would still be preferred on security grounds to our current ssh use) or potentially to a custom script that can do the copy however you like - e.g. (because a host can be specified) via a special data mover machine that sees the file systems on both hosts).

So if I haven't misunderstood any of the above, it seems feasible.

[matthewrmshin]

Tentative cylc-8. Can be pushed back.

[hjoliver]

I'll push this back to some-day @matthewrmshin, because the site that motivated this issue has stopped asking for it (they've accepted our use of ssh).