Nodeup error for docker 18.09.9_ kops upgrade from 15.03 to 16.04_Selinux_Dependency Packages are missing_AMI kernel mismatch.

[kaviarasan-ex2]

1. we see the dependency error in our master when we try upgrade our cluster from 1.15.3 to 1.16.4 while kops tried installing the docker 18.09.9.

Jul 23 10:34:14 <Ip's removed> nodeup: I0723 10:34:14.565660 29510 files.go:100] Hash matched for "/var/cache/nodeup/packages/container-selinux": sha1:7de4211fa0dfd240d8827b93763e1eb5f0d56411
Jul 23 10:34:14 <Ip's removed> nodeup: I0723 10:34:14.591642 29510 files.go:100] Hash matched for "/var/cache/nodeup/packages/docker-ce-cli": sha1:0c51b1339a95bd732ca305f07b7bcc95f132b9c8
Jul 23 10:34:14 <Ip's removed> nodeup: I0723 10:34:14.591680 29510 package.go:304] running command [/usr/bin/rpm -i /var/cache/nodeup/packages/docker-ce /var/cache/nodeup/packages/containerd.io /var/cache/nodeup/packages/container-selinux /var/cache/nodeup/packages/docker-ce-cli]
Jul 23 10:34:14 <Ip's removed> nodeup: W0723 10:34:14.651086 29510 executor.go:130] error running task "Package/docker-ce" (8m48s remaining to succeed): error installing package "docker-ce": exit status 4: warning: /var/cache/nodeup/packages/docker-ce: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY
Jul 23 10:34:14 <Ip's removed> nodeup: warning: /var/cache/nodeup/packages/container-selinux: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Jul 23 10:34:14 <Ip's removed> nodeup: error: Failed dependencies:
Jul 23 10:34:14 <Ip's removed> nodeup: selinux-policy >= 3.13.1-216.el7 is needed by container-selinux-2:2.107-1.el7_6.noarch
Jul 23 10:34:14 <Ip's removed> nodeup: selinux-policy-base >= 3.13.1-216.el7 is needed by container-selinux-2:2.107-1.el7_6.noarch
Jul 23 10:34:14 <Ip's removed> nodeup: selinux-policy-targeted >= 3.13.1-216.el7 is needed by container-selinux-2:2.107-1.el7_6.noarch
Jul 23 10:34:14 <Ip's removed> nodeup: I0723 10:34:14.651125 29510 executor.go:145] No progress made, sleeping before retrying 1 failed task(s)

2. we tried use the latest AMI and we fetched the AMI from kops supported amazon 2linux ami from the kops supported AMI list. As per the page it has been told that the AMI's are kernel 4.19 but where as in the AWS console the AMI details the kernel has been shown that 4.14. Should it be AWS issue or kops issue regarding the kernel mismatch? AMI in refernce that we've got from the below qery output "ami-0732b62d310b80e97"

AMI's have been fetched from as per kops official advise.

https://kops.sigs.k8s.io/operations/images/#amazon-linux-2

Amazon Linux 2¶
Amazon Linux 2 is based on Kernel version 4.19 which fixes some of the bugs present in RHEL/CentOS 7 and effects are less visible.

Available images can be listed using:

aws ec2 describe-images --region ap-south-1 --output table
--owners 137112412989
--query "sort_by(Images, &CreationDate)[].[CreationDate,Name,ImageId]"
--filters "Name=name,Values=amzn2-ami-hvm-2-x86_64-gp2"

3)As part of testing and vulnerability checks we provisioned the machine using the AMI. But we couldn't see the dependency packages in the latest AMI as well.

AMI used: ami-0732b62d310b80e97

[ec2-user@<Ip's removed> ~]$ yum list installed | grep "selinux"
libselinux.x86_64 2.5-12.amzn2.0.2 installed
libselinux-python.x86_64 2.5-12.amzn2.0.2 @amzn2-core
libselinux-utils.x86_64 2.5-12.amzn2.0.2 installed
selinux-policy.noarch 3.13.1-192.amzn2.6.3 @amzn2-core
selinux-policy-targeted.noarch 3.13.1-192.amzn2.6.3 @amzn2-core
[ec2-user@<Ip's removed> ~]$ uname -r
4.14.181-140.257.amzn2.x86_64

[kaviarasan-ex2]

Any update on this please ?

[kaviarasan-ex2]

we're aware of this work around.. But we doesn't want kops to explicitly say to pick a docker version when we're upgrading to 1.16.4 or higher as we're using it in across all our environments. Please advise asap

#8803

But this pull request says it has been resolved and 18.09.9 is supports from >=1.14
#7860
Does this PR introduce a user-facing change?:

Default Docker version will change to 18.09.9 for k8s >= 1.14 and 19.03.4 for k8s >= 1.17.
Users will be able to manually select version 19.03.4.

spec:
docker:
version: 19.03.4

[kaviarasan-ex2]

@hakman can you please help me with resolution here.

[hakman]

You should use Kops 1.18.0-beta.2 for your setups. This should install Amazon Linux 2 without issues.

[kaviarasan-ex2]

@hakman At present our production version is 13.x and 14.x respectively for two different applications respectively. We see there are a lot of changes in the higher versions with respect to API, etc.. To test our application 18.x and that's again with beta, however we need to test our applications in 16.x , 17.x than 18 right.. Moreover using beta in production isn't advisable. We see this issue persist only in amazon 2linux and we're using amzon 2linux in across our environments. Also from the official page https://kops.sigs.k8s.io/operations/images/#amazon-linux-2 amzon 2linux kernel is says 4.19 but where as when we provision the instance we see the kernel is 4.14 only. Is there any possibility other than explicit way of choosing docker 18.06.03 for kops 16.x. Looks like we'll face this issue in 17.x as well. can you please let us know is there any other way that we can mitigate this and let kops choose the docker version while we upgrade the cluster.. It's highly helpful for the change that we're doing..

[hakman]

You can use Kops 1.18 to install Kubernetes 1.11+, so that shouldn't be an issue.
Kops 1.18 will be released shortly if you really want "non beta". Though, if no one tries the "beta", issues will be discovered very late. Keep in mind that support for Amazon Linux 2 was experimental until Kops 1.18.
https://kops.sigs.k8s.io/operations/images/#distro-support

With regard to your issue, if there would have been a solution, would have been fixed some time ago. Unfortunately, there is no way to install Docker CentOS RPMs on Amazon Linux 2. Feel free to try and see if it works now.

That being said, I don't see other way to go, except using Kops 1.18.

[kaviarasan-ex2]

@hakman Thanks for your response and I'm still confused with the below one.

Also from the official page https://kops.sigs.k8s.io/operations/images/#amazon-linux-2 amzon 2linux kernel says 4.19 but where as when we provision the instance we see the kernel is 4.14 only.

[hakman]

When I updated that document I was under the impression that Amazon Linux 2 moved to Kernel 4.19 as default. It's not the case as it's still just optional. I will update the docs soon.
https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-linux-2-extras-provides-aws-optimized-versions-of-new-linux-kernels/

Btw, in case you get to try the new 1.18, I would appreciate some feedback on how Amazon Linux 2 works. Thanks!

[kaviarasan-ex2]

Thanks @hakman That's great! I'll keep the feedback posted as soon as I got a chance to try that out. Please clarify me, does k8s 1.16.x supports docker 18.06.03 ? When we explicitly say kops to install this particular docker version this works for us but we still doesn't know whether this particular docker version supports k8s 1.16x with kops 1.16.4 . Please advise..

[hakman]

You can check the Kubernetes release notes for what Docker versions are supported:
https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG

As to what Kops uses as defaults for various Kubernetes versions, the answers are here:

kops/pkg/model/components/docker.go

Lines 50 to 55 in 2a73c9d

	if b.IsKubernetesGTE("1.17") {
	docker.Version = fi.String("19.03.11")
	} else if b.IsKubernetesGTE("1.16") {
	docker.Version = fi.String("18.09.9")
	} else if b.IsKubernetesGTE("1.12") {
	docker.Version = fi.String("18.06.3")

[kaviarasan-ex2]

@hakman Thanks!