Add upgrade steps, instructions for 2023.9.1 #2029

iameskild · 2023-09-25T00:45:47Z

Reference Issues or PRs

What does this implement/fix?

Supersedes #2021 (@kenafoster I couldn't push to your remote branch or open PR against that branch)

Put a x in the boxes that apply

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds a feature)
Breaking change (fix or feature that would cause existing features not to work as expected)
Documentation Update
Code style update (formatting, renaming)
Refactoring (no functional changes, no API changes)
Build related changes
Other (please describe):

Testing

Did you test the pull request locally?
Did you add new tests?

I tested these upgrade commands (and redeployment) on GCP and Azure. @kenafoster did you happen to test the upgrade command / redeployment on AWS?

Any other comments?

kenafoster · 2023-09-26T13:56:48Z

Tested upgrade on AWS - looks good

Going from 2023.5.1 to 2023.9.1, the cluster was destroyed and rebuilt (@iameskild addressed this with more warnings in the upgrade step to 2023.7.1, where the cluster destroy occurs)
Going from 2023.7.1 to 2023.9.1, worked. I upgraded K8S from 1.24 to 1.26 (note in AWS you must manually upgrade node pools after upgrading the control plane). After setting 1.26 in nebari config, the 2023.9.1 deploy ran successfully. CDS dashboards were present in my original deployment, and the upgrade command removed the config then the deploy removed them from the app

fangchenli · 2023-09-27T17:24:55Z

src/_nebari/constants.py

@@ -8,7 +8,7 @@
 # 04-kubernetes-ingress
 DEFAULT_TRAEFIK_IMAGE_TAG = "2.9.1"

-HIGHEST_SUPPORTED_K8S_VERSION = ("1", "26", "7")
+HIGHEST_SUPPORTED_K8S_VERSION = ("1", "26", "9")


Just a question. How is this determined? Have we tested newer versions?

I have tested this new version on all of the cloud providers and I bumped it so we could support DOKS version 1.26.9-do.0 since this appears to be only version of kubernetes 1.26 available on DO.

src/_nebari/stages/infrastructure/template/gcp/versions.tf

src/_nebari/upgrade.py

aktech · 2023-09-28T20:25:45Z

src/_nebari/utils.py

+    """Return the major.minor version of the k8s version string."""
+
+    k8s_version = str(k8s_version)
+    # Split the input string by the first decimal point
+    parts = k8s_version.split(".", 1)
+
+    if len(parts) == 2:
+        # Extract the part before the second decimal point
+        before_second_decimal = parts[0] + "." + parts[1].split(".")[0]
+        try:
+            # Convert the extracted part to a float
+            result = float(before_second_decimal)
+            return result
+        except ValueError:
+            # Handle the case where the conversion to float fails
+            return None
+    else:
+        # Handle the case where there is no second decimal point
+        return None


you can use packaging for this:

>>> from packaging import version >>> version.parse('2.3.4') <Version('2.3.4')> >>> version.parse('2.3.4') > version.parse('2.3.1') True

The DO versions aren't just numeric (example - 1.18.19-do.0) and I get back packaging.version.InvalidVersion when trying to use this library for them

That's a slug, not the actual k8s versions, you can get explicit kubernetes versions too, e.g.:

$ doctl kubernetes options versions Slug Kubernetes Version Supported Features 1.28.2-do.0 1.28.2 cluster-autoscaler, docr-integration, ha-control-plane, token-authentication 1.27.6-do.0 1.27.6 cluster-autoscaler, docr-integration, ha-control-plane, token-authentication 1.26.9-do.0 1.26.9 cluster-autoscaler, docr-integration, ha-control-plane, token-authentication 1.25.14-do.0 1.25.14 cluster-autoscaler, docr-integration, ha-control-plane, token-authentication

Alternatively you can split by - and use the first part.

src/_nebari/utils.py

iameskild · 2023-09-25T00:53:33Z

src/_nebari/stages/infrastructure/template/gcp/versions.tf

@@ -2,7 +2,7 @@ terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
-      version = "4.83.0"
+      version = "4.8.0"


I needed to revert to what we had in our previous release because when I tried to redeploy with a newer Kubernetes version, it complained about the following although many of those fields are indeed set:

Error: googleapi: Error 400: At least one of ['node_version', 'image_type', 'updated_node_pool', 'locations', 'workload_metadata_config', 'upgrade_settings'] must be specified., badRequest

Otherwise, there doesn't seem away around this unless you delete the node groups and then remove them from the Terraform state which is an accident prone task...

iameskild · 2023-09-25T00:54:22Z

src/_nebari/stages/infrastructure/template/gcp/modules/kubernetes/main.tf

@@ -57,10 +57,6 @@ resource "google_container_cluster" "main" {
    }
  }

-  cost_management_config {


Removed because not supported with GCP Terraform provider version 4.8.0 (see other comment for reason why).

iameskild · 2023-09-25T00:55:36Z

src/_nebari/upgrade.py

@@ -22,6 +27,8 @@
 )
 ARGO_JUPYTER_SCHEDULER_REPO = "https://github.com/nebari-dev/argo-jupyter-scheduler"

+UPGRADE_KUBERNETES_MESSAGE = "Please see the [green][link=https://www.nebari.dev/docs/how-tos/kubernetes-version-upgrade]Kubernetes upgrade docs[/link][/green] for more information."


I created these docs to walk folks through the Kubernetes upgrade process: nebari-dev/nebari-docs#367

I would like to test this for Digital Ocean, I just haven't had the time yet...

I've tested this on Digital Ocean.

iameskild · 2023-09-25T00:56:34Z

tests/tests_unit/test_cli_upgrade.py

+@pytest.mark.skipif(
+    _nebari.upgrade.__version__ < "2023.9.1",
+    reason="This test is only valid for versions <= 2023.9.1",
+)


To get these tests to run, we need to create a new tag 2023.9.1. You can do this locally to confirm they pass.

iameskild · 2023-09-25T00:57:58Z

src/_nebari/upgrade.py

+                        "-> The Kubernetes version is multiple minor versions behind the minimum required version. You will need to perform the upgrade one minor version at a time.  For example, if your current version is 1.24, you will need to upgrade to 1.25, and then 1.26."
+                    )
+                rich.print(
+                    f"-> Update the value of [green]{provider_config_block}.kubernetes_version[/green] in your config file to a newer version of Kubernetes and redeploy."


I can confirm that redeploying with a Kubernetes version one minor higher than the one the user is running will work for GCP and Azure. @kenafoster can you confirm that this will also work as advertised on AWS?

Yes it works on AWS. However, if they do it multiple versions, they'll need to upgrade the node pools manually, and that part must be done outside Nebari

iameskild · 2023-09-27T18:47:44Z

src/_nebari/constants.py

@@ -8,7 +8,7 @@
 # 04-kubernetes-ingress
 DEFAULT_TRAEFIK_IMAGE_TAG = "2.9.1"

-HIGHEST_SUPPORTED_K8S_VERSION = ("1", "26", "7")
+HIGHEST_SUPPORTED_K8S_VERSION = ("1", "26", "9")


I have tested this new version on all of the cloud providers and I bumped it so we could support DOKS version 1.26.9-do.0 since this appears to be only version of kubernetes 1.26 available on DO.

src/_nebari/stages/infrastructure/template/gcp/versions.tf

src/_nebari/upgrade.py

aktech

I don't have any strong objection, that should stop this PR from getting merged, apart from some minor nitpicks. I am happy to get this in as long as tests pass.

thanks for all the work @kenafoster @iameskild

kenafoster and others added 12 commits September 19, 2023 21:03

Upgrade step - remove CDS dashboard config

624dad9

start K8S check action

24ceff4

Add K8S checks

8b61979

Update K8S checks

d8c9679

pre-commit changes

2fda6ab

Ensure GCP k8s is upgraded properly

0e624e8

Merge branch 'develop' into gcp_k8s_upgrade

9036319

Update upgrade command, tests

008d117

Clean up tests, only run if nebari version is available

8c515e2

Clean up docs message

d046d58

Update message

d7df1bb

Uncomment assert

33198a7

iameskild added area: user experience 👩🏻‍💻 impact: high 🟥 This issue affects most of the nebari users or is a critical issue project: JATIC Work item needed for the JATIC project labels Sep 25, 2023

iameskild changed the title ~~Gcp k8s upgrade~~ Add upgrade steps, instructions for 2023.9.1 Sep 25, 2023

Merge branch 'develop' into gcp_k8s_upgrade

8bbbc29

iameskild mentioned this pull request Sep 25, 2023

2023.9.1 upgrade #2021

Closed

6 tasks

Add skipif to test_cli_upgrade_to_2023_9_1_cdsdashboard_removed

a7762aa

iameskild added this to the Release 2023.9.1 milestone Sep 25, 2023

iameskild added 3 commits September 25, 2023 17:25

Add DANGER warning for AWS, clean up

aae93a9

Add warning for AWS to 2023.7.2 upgrade

89ede8f

Add message, prevent_deploy, tests for 2023.7.1

3c333d2

pavithraes mentioned this pull request Sep 26, 2023

How to upgrade kubernetes nebari-dev/nebari-docs#367

Merged

18 tasks

Merge branch 'develop' into gcp_k8s_upgrade

59c7292

fangchenli reviewed Sep 27, 2023

View reviewed changes

aktech reviewed Sep 28, 2023

View reviewed changes

iameskild commented Sep 29, 2023

View reviewed changes

2023.9.1 AWS upgrade changes (#2041)

d6dd63b

[pre-commit.ci] Apply automatic pre-commit fixes

5d8cde4

iameskild requested a review from aktech September 29, 2023 03:33

aktech reviewed Sep 29, 2023

View reviewed changes

aktech approved these changes Oct 2, 2023

View reviewed changes

iameskild merged commit 056b420 into develop Oct 2, 2023
25 checks passed

iameskild deleted the gcp_k8s_upgrade branch October 2, 2023 16:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add upgrade steps, instructions for 2023.9.1 #2029

Add upgrade steps, instructions for 2023.9.1 #2029

iameskild commented Sep 25, 2023

kenafoster commented Sep 26, 2023

fangchenli Sep 27, 2023 •

edited

Loading

iameskild Sep 27, 2023

aktech Sep 28, 2023

kenafoster Sep 29, 2023

aktech Sep 29, 2023

iameskild Sep 25, 2023

iameskild Sep 25, 2023

iameskild Sep 25, 2023

iameskild Sep 27, 2023

iameskild Sep 25, 2023

iameskild Sep 25, 2023

kenafoster Sep 29, 2023

iameskild Sep 27, 2023

aktech left a comment

Add upgrade steps, instructions for 2023.9.1 #2029

Add upgrade steps, instructions for 2023.9.1 #2029

Conversation

iameskild commented Sep 25, 2023

Reference Issues or PRs

What does this implement/fix?

Testing

Any other comments?

kenafoster commented Sep 26, 2023

fangchenli Sep 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aktech left a comment

Choose a reason for hiding this comment

fangchenli Sep 27, 2023 •

edited

Loading