Generate authentication keys

[datamel]

I suspect this PR will result in further changes to code so I will squash once fully ready to be merged.

Keys are now all stored in the .service directory of the suite rather than the client keys atored in the .cylc directory.

Keys are no longer regenerated - cylc errors are raised instead

This has been left extensible for future multiple platform key generation.

Closes #3444
Closes #3445

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Does not contain off-topic changes (use other PRs for other changes).
• Appropriate tests are included (unit and/or functional).
• No change log entry required - invisible to users.
• Documentation PR to be competed authentication: wash up #3449

[kinow]

Not really a proper review. Just quickly read it to know more or less what's changing. But great improvements with f-strings, and also good unit tests! 👏

[oliver-sanders]

Looks good, a few small comments which should be quick fixes. Namely we need to swap the order in which we set permissions and move files to make sure they don't become vulnerable in between.

[hjoliver]

(@dpmatthews plans to take a close look, and do some testing, before others pitch in here)

[datamel]

@dpmatthews and I have just been through this together. I think it is now ready for review, @hjoliver I will add you to the list of reviewers. Thank you to all for the comments so far.
Note that this PR will also close #3444 in addition to #3445.
I think it is all now working as expected and remote authentication is now working too.
There is a separate issue I will raise to ensure errors are recorded correctly when keys are missing remotely.
The authentication keys are now all stored in: <SUITE_NAME>/.service. Client public keys have their own sub directory: <SUITE_NAME>/.service/client_public_keys.
There is still much legacy code to remove. This will be removed in ticket: authentication: wash up #3449.
I will also rebase/squash once everyone is happy and it is ready for merge.

[dwsutherland]

LGTM 👍

Works end-to-end WFS <=> UIS <=> UI

2020-01-24T17:01:08+13:00 DEBUG - Starting
2020-01-24T17:01:08+13:00 DEBUG - auth received API command b'CURVE'
2020-01-24T17:01:08+13:00 DEBUG - Configure curve: *[/home/sutherlander/cylc-run/baz/.service/client_public_keys]
.
.
.
2020-01-24T17:01:47+13:00 DEBUG - version: b'1.0', request_id: b'1', domain: '', address: '127.0.0.1', identity: b'', mechanism: b'CURVE'
2020-01-24T17:01:47+13:00 DEBUG - ALLOWED (CURVE) domain=* client_key=b'@:LiHf*406UmxVmgT1dyY.tZ3gN3:XhZxXv?/i{Q'
2020-01-24T17:01:47+13:00 DEBUG - ZAP reply code=b'200' text=b'OK'
2020-01-24T17:02:11+13:00 DEBUG - version: b'1.0', request_id: b'1', domain: '', address: '127.0.0.1', identity: b'', mechanism: b'CURVE'
2020-01-24T17:02:11+13:00 DEBUG - ALLOWED (CURVE) domain=* client_key=b'@:LiHf*406UmxVmgT1dyY.tZ3gN3:XhZxXv?/i{Q'
2020-01-24T17:02:11+13:00 DEBUG - ZAP reply code=b'200' text=b'OK'
2020-01-24T17:02:11+13:00 INFO - [client-command] release_suite sutherlander@cortex-vbox:cylc-release
.
.
.

Definitely some improvements ... Will have another read and try to think about it in the wider context.

[oliver-sanders]

Code looks good.

Tested as working including remote functionality 🚀

Cylc creates the two keys we expect it to; client (cli) and server (ser) both in public (pub) and private (pri) variants:

• @localhost ~/cylc-run/suite/.service/
  • cli-pri client.key_secret
  • cli-pub client_public_keys/client.key
  • ser-pri server.key_secret
  • ser-pub server.key

On the remote hosts Cylc copies two keys across:

• @remotehost ~/cylc-run/<suite>/.service/
  • cli-pri client.key_secret
  • ser-pub server.key

(I find the filenames curve/zmq uses for these keys really darned confusing).

Note: Copying the private client key is bad security but this is set aside for future work #3443

[oliver-sanders]

A quick rebase and we are good to go!

[dpmatthews]

Remote tests now working fine for me