[ENH] - Explore ways of automating nebari upgrades

[costrouc]

Feature description

Currently nebari upgrades are performed as described in https://www.nebari.dev/docs/how-tos/nebari-upgrade/. It would be nice to have more gitops(y) approach to upgrading clusters.

Value and/or benefit

Making upgrades easier.

Anything else?

My proposal would be to add additional workflows to the github-actions using workflow dispatch https://docs.github.com/en/actions/using-workflows/manually-running-a-workflow. We should provide options in the upgrades e.g. whether to upgrade images in the yaml config.

[iameskild]

Often during the upgrade step, there are times when we require some user input. At the moment they come in a few forms:

1. asking users to accept particular changes to the nebari-config.yaml, such as do you want upgrade to the latest jupyterlab image?
2. asking user to manually delete or remove particular kubernetes resources, such as delete the old Helm-deployed CRDs
3. asking user to perform a manual back because the upgrade is destructive in one way or another

For item 1, we can include an input form as part of the workflow_dispatch asking them which keys they are fine upgrading (ie jupyterhub image tag but not jupyterlab).

For item 2, this might also require us to update the upgrade command so to directly interact with the kubernetes cluster (via the kubernetes python client).

And item 3 requires an automated backup feature 😅

[costrouc]

@iameskild I agree on (2) and (3) being hard to automate/not possible.

I'd like to scope this issue on how we can have a triggered workflow for updates with just a few options. I think this is achievable and would not require much work.

As of now it looks like the only option we'd see is y/n update the images?

[iameskild]

That sounds good to me! I think starting with either all Yes or all No for the upgrade options makes sense.

[costrouc]

Steps we propose:

1. User visits github-actions tab
   

2. User visits the upgrade workflow we have created

3. User will trigger a workflow

Behind the scenes this will trigger a github actions workflow that will:

• update the nebari-config.yaml within CI
• re-commit the config changes back to the repository

[iameskild]

From the user's perspective, when they need to perform an upgrade:

GITLAB

1. Run a pipeline manually (Build > Pipeline > Run Pipeline)
2. Enter a variable for: desired Nebari version and accept or decline updates to docker image tags
3. Pipeline runs and performs upgrade automatically, merging the updated files back into the repo

GITHUB

1. Run a workflow manually (as shown in the images in the comment above)
2. Enter the variables for: desired Nebari version and accept or decline updates to docker image tags
3. Workflow runs and performs upgrade automatically, merging the updated files back into the repo

[kalpanachinnappan]

Hi @iameskild!

Are the manual steps for each upgrade going to be a 100% required? e.g. removing some K8s resources.

Would it be possible to trigger an update by editing specific settings in nebari-config.yaml?
E.g. Editing nebari_version would trigger the update to happen in CI when nebari deploy runs.

[iameskild]

Hi @kalpanachinnappan, at the moment the upgrade command modifies the nebari-config.yaml, we then need to render those changes (ie updates to the stages , CI/CD workflow files). I think what you're asking for is possible but would require a new workflow / pipeline that performs all of these steps in order:

This workflow would need to be triggered on any updates to the nebari_version key in the nebari-config.yaml:

1. Run nebari upgrade to upgrade the nebari-config - we will need to build in a means to auto accept or auto decline the changes the upgrade command is asking about (perhaps we can repurpose the --attempt-fixes flag which already exists).
2. Run nebari render to render all of the changes included in the new version.
3. Run nebari deploy to deploy those changes and merge the changes back into main

A few things worth considering:

• what to do when the nebari_version is a from development branch? pip install git+https://...
• what to do when there are manual changes needed during the upgrade?