File installation via rsync and install target installation added

[datamel]

Install by target

Remote installation is now on a per install target rather than a per platform basis. Targets can be configured in the global.cylc e.g.

[platforms]
    [[<platform name>]]
        install target = localhost

• Remote initialisation is skipped where installed target is local.
• Remote inititalisation returns REMOTE_INIT_FAILED where a key for a different target is found on the remote.
• If a key for the correct target is found, it refreshes the keys. This was to prevent a situation where you could not run an existing suite that was cancelled inelegantly e.g. ctrl-c.

Existing keys are identified by filename (where the install target is now added to it). The initial implementation was through the key metadata. However, it turns out that metadata goes in the secret key only and not the public key.

There is now no mechanism for checking for a shared file system because uuid file has been removed. This now has to be configured in global.cylc.

Notes on call to task_remote_mgr.remote_init(...)

Previously, platform name was being sent to remote_init and then a skeleton platform dictionary was generated from that. We're now sending a complete dictionary rather than regenerating.

In restart_remote_init we were using a set to pick distinct platforms. As platform is a dictionary and not an object (where we could add a hash), the method is reworked to create a list of platforms filtered as distinct install targets

File install via Rsync

File installation is now via rsync. Files to be installed are configurable in the flow.cylc file (translated to rsync include and filter rules).

File install is still a push but happens after remote initialisation in the callback because we need to know whether or not the file install is required before we push files across.

Add files to the rsunc in the flow.cylc:

[scheduler]
    includes = dir/, dir2/, file1, file2

This currently supports top level directories and files - I have also added validation for this feature.

Rsync Logging

New RSYNC_LOG log which will be created in the suite log directory. Any errors from the file installation will appear in the suite log.

These changes close #3687

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase). - I'll tidy branch after feedback.
• Does not contain off-topic changes (use other PRs for other changes).

• Appropriate tests are included (unit and/or functional).
• Existing tests also adapted.

• Appropriate change log will be completed after feedback

• (master branch) I will open a documentation PR at cylc/cylc-doc.
• No dependency changes.

[datamel]

Note, I will do a read through/double check doc strings and fix a broken functional test that I have written (tests/f/remote/03-install-target.t)

[oliver-sanders]

Looks good, some RST formatting comments...

[oliver-sanders]

Finished code review, some minor comments:

[oliver-sanders]

The new logging is working fine, logs roll over when the suite restarts.

One small niggle is that the log files are created a few seconds apart so their timestamps don't match but we can address that later on.

$ ls -1tr log/suite/
file-installation-log.20200923T112819+01
log.20200923T112811+01
log
file-installation-log
file-installation-log.20200923T113034+01
log.20200923T113027+01

[datamel]

One small niggle is that the log files are created a few seconds apart so their timestamps don't match but we can address that later on.

$ ls -1tr log/suite/
file-installation-log.20200923T112819+01
log.20200923T112811+01
log
file-installation-log
file-installation-log.20200923T113034+01
log.20200923T113027+01

Yes, that is a bit of a niggle. I guess it would be nicer for the suite log and the file-installation-log timestamps to match. I can put an issue up and assign myself so it doesn't hold this up if you like?

[oliver-sanders]

Tests pass for me (with the cylc version diff I sent you), looks good, works as expected.

Few more comments which could be addressed on this PR if time permits but which needn't delay merge.

[hjoliver]

A bunch of minor comments and suggestions, but generally looking great 🎉

[wxtim]

The conflicting files

 tests/functional/validate/71-platform-basic.t
tests/functional/validate/72-host-to-platform-upgrade.t 

have been moved to tests/functional/platforms by me in the meantime, so the rebase should be easy, just delete to match master. :)

[datamel]

Thanks @wxtim!

[hjoliver]

🎉 nice work @datamel