setup: migrate to jupyter server 2

[oliver-sanders]

Closes #411

Migrate from Jupyter Server 1-2.

There has been a rejig of the get_current_user logic so this should be accompanied by a security check of authorisation/authentication.

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• CHANGES.md entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc hub: document multi-user config post jupyter server v2 upgrade cylc-doc#625.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.
• Security penetration test.

[codecov-commenter]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.28 🎉

Comparison is base (929714a) 76.96% compared to head (32936f7) 77.25%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #450      +/-   ##
==========================================
+ Coverage   76.96%   77.25%   +0.28%     
==========================================
  Files          12       12              
  Lines        1385     1407      +22     
  Branches      236      236              
==========================================
+ Hits         1066     1087      +21     
- Misses        276      277       +1     
  Partials       43       43              

Impacted Files	Coverage Δ
cylc/uiserver/handlers.py	86.53% <100.00%> (+1.12%)	⬆️

... and 5 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[mahendrapaipuri]

Any chances this will be merged soon? We would like to run UI server as a JupyterLab extension in our JupyterHub deployment and the current constraint on jupyter_server version is a major blocking point for us.

[hjoliver]

Yes, it's ready for review now, and not a big change. It should be merged pretty soon.

[hjoliver]

Looks good at first glance. I haven't tested it yet.

[oliver-sanders]

@mahendrapaipuri, we should be able to get this reviewed soonish, however, this feature is scheduled for release along with Cylc 8.2.0 so won't be released until some other work is completed.

If you want to get going with Jupyter Server 2 and Cylc 8.1, we might be able to provide you with a development branch you could install from to save waiting. Let us know if that would help.

[mahendrapaipuri]

Cheers @oliver-sanders!! Well, we are not in a rush. I will keep track of this PR and once it gets merged I can test using the master branch on our platform.

[oliver-sanders]

Great, just be aware you'll need to install cylc-flow from master too.

[MetRonnie]

Still need to update to >=3.8

Or should it be >=3.8,<3.10?

cylc-uiserver/setup.cfg

Line 43 in 1446760

python_requires = >=3.7

[oliver-sanders]

sorry, didn't read properly, done.

[MetRonnie]

LGTM, I've done a basic test of running the UI after pip installing on this branch

[wxtim]

• Look at code
• Have a good play with Cylc Hub
• Pen test

[oliver-sanders]

[Update] Apologies, this is a little more complex than it initially looked due to changes in authorisation and jps 2.7.

[oliver-sanders]

Ok, this is now looking ok but still needs a little work.

This PR now upgrades us, both to Jupyter Server v2 and to newer versions of Jupyter Hub which has required some re-working of the authorisation which will need to be carefully examined and tested. The authorisation model requires recent versions of both for multi-user access, but I haven't flushed out the full ranges yet.

The good news is that recent Jupyter Hub versions have added a scope to authorise the starting of other users servers so this improves multi-user functionality greatly.

Along with the documentation PR:

• Partially addresses Running JupyterLab alongside the Cylc UI #298
• Closes hub: investigate spawning other users UI servers #275

TODO:

• Fix tests.
• Check jupyter_server version range (we may need to up the minimum).
  • I've gone with a minumum of 2.7, if this is problematic, there's the potential to reduce this slightly with further research and testing.
• Decide the jupyter_hub version range (this may require very recent releases).
  • I've gone with a minumum of 4.0, if this is problematic, there's the potential to reduce this slightly with further research and testing.
• Further penetration testing of Cylc UIServer and Jupyter Lab according to the documented configurations.
• Add details in the changes section of the docs.

Stale

[MetRonnie]

Done another review. No comments at the moment, pending changes

[oliver-sanders]

Got the tests passing. It wasn't possible to keep the unauthenticated tests because the jupyter_server test framework token authenticates all requests, turning this off requires in-depth hacking and doesn't really test the implementation properly.

The unauthorised userprofile test has been removed as the userprofile no longer requires authorisation since it only gives you information about yourself.

[MetRonnie]

Tested cylc gui and cylc hub, both working

[hjoliver]

LGTM, tested as working.