WIP - Support non-standard remote run directories.

[hjoliver]

Works with a non-default cylc global config "run directory" setting, e.g:

# global.rc
[hosts]
   [[server1]]
       run directory = /nfs/home1/$USER

This can be useful if, for instance, your $HOME on the cylc remote server is not the same as $HOME on the compute cluster that it fronts (but the compute home dir is still mounted on the remote). We need the suite run directory to be set up where the jobs see it.

TODO -

• test with default and non-standard run directory on a remote host
• handle the similar work directory setting too (for share and work location)
• with rose root-dir symlinks
• with rose root-dir{share/work} symlinks
• test with --new (not working?)
• document
• testing?

[matthewrmshin]

You probably want to concentrate on getting the --remote=... option set up correctly when rose suite-run relaunches itself with --remote=... in the job host. (And sorry for the archaic code.)

[hjoliver]

You probably want to concentrate on getting the --remote=... option...

I used new --run-dir=DIR and --work-dir=DIR options for the remote invocation, which seems to be fine because the command help is not defined by the actual options (i.e. the new options are not exposed to users). But should I be passing these values to the remote side with --remote... instead??

[hjoliver]

Travis CI failure is only because pygtk.org has evidently disappeared off the internet.

Warning, treated as error:
/home/travis/build/metomi/rose/sphinx/api/rose-gtk-library.rst.rst:13:broken link: http://www.pygtk.org/ (HTTPConnectionPool(host='www.pygtk.org', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2b5643bbaad0>: Failed to establish a new connection: [Errno -2] Name or service not known',)))
make: *** [linkcheck] Error 2
make: Leaving directory `/home/travis/build/metomi/rose/sphinx'

[hjoliver]

(Tested successfully on two networked VMs, and on the Azure cluster with PBS that mimics the BoM clusters; now awaiting testing at BoM...).

[matthewrmshin]

Yes, attempts to resolve www.pygtk.org are returning Name or service not known. (Time to say goodbye to PyGTK?)

[matthewrmshin]

I don't really mind either way - given that this logic will migrate to Cylc soon!

The --remote=key1=value1,key2=value2,... settings are split by the _run_remote method in the r_opts dict - which is supposed to make life easier. (It was also done like this to avoid too many CLI options being introduced that are for internal use only).

[matthewrmshin]

Just restarted the build again.

[hjoliver]

https://www.pygtk.org now points at the PyGObject site.

[hjoliver]

@matthewrmshin - I've just retested this again along with cylc/cylc-flow#2877 (i.e. the two parts of cylc/cylc-flow#2779). It all works fine; can you advise on what more needs doing to get this merged to Rose master (specifically tests I guess) and whether next release is feasible.

[DamianAgius]

@matthewrmshin I believe at this point, suite_dir_root is Rose config item root-dir, and suite_name is appropriately set - however neither of these add {pre,suf}fix 'cylc-run' - so if opts.run_dir is set, suite_dir_real = <rose root-dir>/<suite-name> (no /cylc-run/ in between) which is then symlinked to ~/cylc-run which is the exact behaviour I'm seeing.

[matthewrmshin]

@hjoliver @DamianAgius If I remember correctly via email exchanges @DamianAgius has a trick in place to make this work at his site. Has/can it be integrated with this branch?

[hjoliver]

Sorry for the delay - this all went down while I was away on leave. @DamianAgius - I'm now a bit hazy on the details, except that it seems surprising that this worked properly on the Azure nodes that were supposed to mimic your multi-host configuration. I definitely checked those test results many times and would have noticed if the "cylc-run" path component was missing. Now the Azure nodes have been taken down so I can't retest. I wonder if it was a configuration issue - maybe I (incorrectly?) specified the "cylc-run" path in my home dir config, and you didn't?

[matthewrmshin]

@hjoliver Do you still want to pursue this?

[DamianAgius]

@matthewrmshin We are running with the locally patched version of Hilary's branch. I currently have no time to spend on this, but would really appreciate the functionality in master (at some stage). My current workload should ease in the coming weeks, and I'll hopefully have some time then.

[hjoliver]

@hjoliver Do you still want to pursue this?

As Damian (BoM) notes - yes!

[hjoliver]

Next step is, I think, to get feedback from Damian on what was wrong with this branch (causing missing "cylc-run" path component) in his environment, that he managed to fix - when it seemingly wasn't wrong on the simultated environment we had in the cloud ... then I need to figure out how to test it again without said cloud environment ... Docker might save us there.

Doesn't sound like a major hurry to get it into master though, from what @DamianAgius says.

[hjoliver]

(May have been incorrect use of the path config by one of us, probably me).

[hjoliver]

I just rediscovered this ancient PR. @DamianAgius are you still out there? Is this still something you'll need to deal with under Cylc 8 (which is going to be deployed on new hardware, I think?).

[DamianAgius]

Hi @hjoliver - good to hear from you, and hello to the Cylc/Rose team(s).
I cannot confirm at this stage (but do hope!) that this 'feature' will be required as we have not yet finished configuration and testing.

[oliver-sanders]

Closing this old PR now, if there's any followup please comment here.