Adding a CI step to test the upgrade path

[betatim]

Today we (@betatim and @gedankenstuecke) attempted to upgrade a JupyterHub from v0.7-c491965 to v0.7-560a7cd in "one step".

The conclusion after some attempts was that we couldn't do it and decided to helm delete ... and then helm upgrade --install ... Which was Ok in this instance but I was surprised that even with some digging and understanding of the chart my conclusion was that it would require several kubectl edit ... commands to do this.

This is the PR for the attempted update which includes the chart and other setup. I think most of it is a fairly vanilla helm chart setup.

Proposal: we add a step to our CI that installs the last stable release (or the release from N months ago, something we define) with a "typical" values.yaml and then attempts to upgrade it to the latest commit. This would make people sleep better at night because they know there is an upgrade path. Would it slow down our development too much?

[consideRatio]

I'd love this! I've added a "typical values.yaml" for the tools/lint.py file already btw, in #758 I've made use of that for linting and validation of yaml files a part of the CI.

I'm sooooo motivated to get a stable release for JupyterCon, this would be very useful!

Related - About troubleshooting these issues related to helm getting stuck in a bad state where multiple revisions of a release is considered to be Deployed.
#638.

[minrk]

Such a test should also cover upgrading while there are running users to make sure that the updated Hub doesn't lose track of users and proxy routes, etc. Not every Hub version upgrade will allow this, but if it doesn't work we should know about it and advertise it loudly.

[manics]

Is this sufficiently covered by #1427 ?

[consideRatio]

Yes @manics at least now! I captured the idea of checking for upgrading while there are running users to make sure that the updated hub doesnt loose track of users and proxy routes, etc. like Min described!

❤️ @manics, I'm really happy with all the work you put in, you are greatly appreciated by me :)