Kubernetes Version 1.11.8

[tabern]

Update to EKS platform version for Kubernetes 1.11 to use patch version 1.11.8

[mbelang]

@tabern When is that schedule. I absolutely need 1.11.7 to fix an issue with nlb. Thank you.

[whereisaaron]

Seems like 1.11.8 is already available @mbelang. I notice the EKS 1.11 node pools I created yesterday are deploying as 1.11.8-eks.
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.8-eks-7c34c0", GitCommit:"7c34c0d2f2d0f11f397d55a46945193a0e22d8f3", GitTreeState:"clean", BuildDate:"2019-03-01T22:49:39Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

[mbelang]

@whereisaaron, I did notice that when I looked as well. I didn't know the master updates where automated which is nice! Though my node are still 1.11.5 because the ami I used is 1.11.5? Is there a 1.11.8 image available or I am missing something?

[geerlingguy]

$ kubectl version
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.8-eks-7c34c0", GitCommit:"7c34c0d2f2d0f11f397d55a46945193a0e22d8f3", GitTreeState:"clean", BuildDate:"2019-03-01T22:49:39Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Yeah, so it was silently upgraded sometime in the past day or two, it seems (this is an old cluster that started at 1.10 and I manually upgraded to 1.11 a month or so ago).

[mbelang]

Yes I get that the minors are manual and the fix are automated. But I have hard time finding the corresponding ami for the nodes that contains 1.11.8

[kurtostfeld]

EKS Platform Version documents new 1.11 version eks.2 with Kubernetes 1.11.8:
https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html

The latest documented Amazon EKS-optimized AMI images are still 1.11.5. The hashes haven't changed in a while:
https://docs.aws.amazon.com/eks/latest/userguide/launch-workers.html

If you do aws eks describe-cluster --name <name> you will see the new 1.11 eks.2

If you do kubectl version, you will see server version "v1.11.8-eks-7c34c0".

If you do kubectl get nodes, you will see node version "1.11.5"

I presume 1.11.8 support isn't complete, which is why this ticket is still open.

[whereisaaron]

What has shipped @tabern? Just the control plane or have the worker AMI's been updated? @kurtostfeld thought not yet? Are there release notes or a blog entry somewhere?

[mbelang]

Seems that 2 days ago when I took the latest AMI, it was still 1.11.5 as @kurtostfeld and I mentioned above. My clusters were indeed 1.11.8.

[kurtostfeld]

fyi, nothing has changed since my last comment.

I just created a new EKS cluster from scratch just now (on 2019-03-15) (using Terraform to help).

aws eks describe-cluster --name <name> shows "version": "1.11" and "platformVersion": "eks.2"

kubectl version shows server version "v1.11.8-eks-7c34c0".

kubectl get nodes shows node version "1.11.5"

I am using launch configuration AMI ID "ami-0c28139856aaf9c3b"

which is the current documented AMI ID for my region us-west-2 from https://docs.aws.amazon.com/eks/latest/userguide/launch-workers.html

If I install prometheus-operator, the "KubeVersionMismatch" alert activates due to the 1.11.5 vs 1.11.8 mismatch.

[dgamo]

Same here. We have a few clusters where master nodes are already on 1.11.8. But there is no AMI available for it. Our prometheus-operator is also firing alams KubeVersionMismatch due the version mismatch.

It seems that the AMI was updated to point to 1.11.8 binaries 4 days ago but it hasn't been published yet, I guess is coming soon... Shall it be considered as "done" ? @tabern

awslabs/amazon-eks-ami@6090f20

[whereisaaron]

I think it is ok to mark things 'done' here if the team have finished their work. But I'd prefer they close tickets with a comment that doesn't leave us in the dark, something like 'Done, 1.11.8 control plane live now, AMIs will appear in the next 5-10 days', or 'Done, 1.11.8 control plane live now, we'll update you here when the AMI's are ready'. Instead we are all confused, wondering if maybe the AMI's are there somewhere and we just can't find them? Maybe they have come out and just the web page hasn't been updated? Maybe they are still coming?

[tabern]

Reopening this. We updated the EKS control plane to kubernetes 1.11.8 last week to deliver a fix for the recent Golang CVE (kubernetes/kubernetes#73379) to the Kubernetes API server. We are working on delivering updated AMIs for EKS worker nodes and will update this issue when they are available.

@whereisaaron this is a good suggestion, we'll communicate patch updates in this way next time.

[geerlingguy]

(Also in the interest of my own education, I re-read the Kubernetes Version Skew documentation, and it does state that running the master kube apiserver at a version up to one minor release newer than kubelet is acceptable and supported.)

[aspekt112]

New AMI appears at Community AMIs:

amazon-eks-node-1.11-v20190327 - ami-06640f41bbd6653bb
EKS Kubernetes Worker AMI with AmazonLinux2 image (k8s: 1.11.9, docker:18.06)

But it has k8s: 1.11.9 and current version of EKS is 1.11.8. Is it correct, that the Removed ulimit reduction issue, #206 solved in the AMI?
Can i update my worker node groups safely?

[llamahunter]

okaaay.... what's the schedule for the control plane to get updated to 1.11.9?

[tabern]

The latest EKS worker node AMI is now version 1.11.9 – docs

The control plane is 1.11.8. Worker nodes are a patch version ahead of the control plane in order to implement the fix for Kubernetes Security Issues (CVE-2019-1002101 and CVE-2019-9946) which does not impact the control plane.

Patch version skew is normal and does not affect the stability of a cluster.

[morganchristiansson]

1.12 was just released too https://aws.amazon.com/about-aws/whats-new/2019/03/amazon-eks-now-supports-kubernetes-version-1-12-and-cluster-vers/