Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error with the amazon-k8s-cni container after the update #689

Closed
cp38510 opened this issue Jul 2, 2021 · 1 comment
Closed

error with the amazon-k8s-cni container after the update #689

cp38510 opened this issue Jul 2, 2021 · 1 comment

Comments

@cp38510
Copy link

cp38510 commented Jul 2, 2021 •
edited
Loading

What happened:
perhaps this is a coincidence, but after updating from ami version amazon-eks-node-1.19-v20210322 to version
amazon-eks-node-1.19-v20210628(ami-0c2ca9cd067f101bc) in region eu-west-3, after some time, an error with container amazon-k8s-cni:v1.7.5-eksbuild.1 appeared on one of the 9 nodes:

Type     Reason     Age                 From                                                Message
  ----     ------     ----                ----                                                -------
  Normal   Scheduled  3m4s                default-scheduler                                   Successfully assigned kube-system/aws-node-n9dmj to ip-10-10-10-10.eu-west-3.compute.internal
  Normal   Pulled     3m3s                kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Successfully pulled image "602401143452.dkr.ecr.eu-west-3.amazonaws.com/amazon-k8s-cni:v1.7.5-eksbuild.1" in 127.941902ms
  Normal   Pulling    3m3s                kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Pulling image "602401143452.dkr.ecr.eu-west-3.amazonaws.com/amazon-k8s-cni-init:v1.7.5-eksbuild.1"
  Normal   Pulled     3m3s                kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Successfully pulled image "602401143452.dkr.ecr.eu-west-3.amazonaws.com/amazon-k8s-cni-init:v1.7.5-eksbuild.1" in 146.91813ms
  Normal   Created    3m3s                kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Created container aws-vpc-cni-init
  Normal   Started    3m3s                kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Started container aws-vpc-cni-init
  Normal   Started    3m2s                kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Started container aws-node
  Warning  Unhealthy  2m56s               kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Readiness probe failed: {"level":"info","ts":"2021-07-02T09:20:59.792Z","caller":"/usr/local/go/src/runtime/proc.go:203","msg":"timeout: failed to connect service \":50051\" within 1s"}
  Warning  Unhealthy  2m46s               kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Readiness probe failed: {"level":"info","ts":"2021-07-02T09:21:09.775Z","caller":"/usr/local/go/src/runtime/proc.go:203","msg":"timeout: failed to connect service \":50051\" within 1s"}
  Warning  Unhealthy  2m36s               kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Readiness probe failed: {"level":"info","ts":"2021-07-02T09:21:19.771Z","caller":"/usr/local/go/src/runtime/proc.go:203","msg":"timeout: failed to connect service \":50051\" within 1s"}
  Warning  Unhealthy  2m26s               kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Readiness probe failed: {"level":"info","ts":"2021-07-02T09:21:29.762Z","caller":"/usr/local/go/src/runtime/proc.go:203","msg":"timeout: failed to connect service \":50051\" within 1s"}
  Warning  Unhealthy  2m16s               kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Readiness probe failed: {"level":"info","ts":"2021-07-02T09:21:39.770Z","caller":"/usr/local/go/src/runtime/proc.go:203","msg":"timeout: failed to connect service \":50051\" within 1s"}
  Warning  Unhealthy  2m6s                kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Readiness probe failed: {"level":"info","ts":"2021-07-02T09:21:49.762Z","caller":"/usr/local/go/src/runtime/proc.go:203","msg":"timeout: failed to connect service \":50051\" within 1s"}
  Warning  Unhealthy  116s                kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Readiness probe failed: {"level":"info","ts":"2021-07-02T09:21:59.769Z","caller":"/usr/local/go/src/runtime/proc.go:203","msg":"timeout: failed to connect service \":50051\" within 1s"}
  Warning  Unhealthy  112s                kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Liveness probe failed: {"level":"info","ts":"2021-07-02T09:22:03.775Z","caller":"/usr/local/go/src/runtime/proc.go:203","msg":"timeout: failed to connect service \":50051\" within 1s"}
  Warning  Unhealthy  106s                kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Readiness probe failed: {"level":"info","ts":"2021-07-02T09:22:09.766Z","caller":"/usr/local/go/src/runtime/proc.go:203","msg":"timeout: failed to connect service \":50051\" within 1s"}
  Normal   Killing    92s                 kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Container aws-node failed liveness probe, will be restarted
  Warning  Unhealthy  86s (x4 over 102s)  kubelet, ip-10-10-10-10.eu-west-3.compute.internal  (combined from similar events): Readiness probe failed: {"level":"info","ts":"2021-07-02T09:22:29.784Z","caller":"/usr/local/go/src/runtime/proc.go:203","msg":"timeout: failed to connect service \":50051\" within 1s"}
  Normal   Pulling    82s (x2 over 3m3s)  kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Pulling image "602401143452.dkr.ecr.eu-west-3.amazonaws.com/amazon-k8s-cni:v1.7.5-eksbuild.1"
  Normal   Created    81s (x2 over 3m3s)  kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Created container aws-node
  Normal   Pulled     81s                 kubelet, ip-10-10-10-10.eu-west-3.compute.internal  Successfully pulled image "602401143452.dkr.ecr.eu-west-3.amazonaws.com/amazon-k8s-cni:v1.7.5-eksbuild.1" in 504.321983ms

logs:

{"level":"info","ts":"2021-07-02T09:14:11.287Z","caller":"entrypoint.sh","msg":"Install CNI binary.."}
{"level":"info","ts":"2021-07-02T09:14:11.305Z","caller":"entrypoint.sh","msg":"Starting IPAM daemon in the background ... "}
{"level":"info","ts":"2021-07-02T09:14:11.307Z","caller":"entrypoint.sh","msg":"Checking for IPAM connectivity ... "}

restarting and recreating the container didn't help, the problem was solved only by deleting the node.

How to reproduce it (as minimally and precisely as possible):
update the ami to the version amazon-eks-node-1.19-v20210628(ami-0c2ca9cd067f101bc)

Anything else we need to know?:
previously, there were no such errors, so I decided to create a ticket

Environment:

  • AWS Region: eu-west-3
  • Instance Type(s): t3a.2xlarge
  • EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.4
  • Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.19
  • AMI Version: amazon-eks-node-1.19-v20210628(ami-0c2ca9cd067f101bc)
  • Kernel (e.g. uname -a): Linux ip-10-10-10-10.eu-west-3.compute.internal 5.4.117-58.216.amzn2.x86_64 Template is missing source_ami_id in the variables section #1 SMP Tue May 11 20:50:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  • Release information (run cat /etc/eks/release on a node):
BASE_AMI_ID="ami-0136f7c838fded2f6"
BUILD_TIME="Mon Jun 28 16:39:26 UTC 2021"
BUILD_KERNEL="5.4.117-58.216.amzn2.x86_64"
ARCH="x86_64"
@cp38510
Copy link
Author

cp38510 commented Jul 14, 2021

the same error was repeated with amazon-eks-node-1.19-v20210628
so it is not related to the eks ami update, similar to aws/amazon-vpc-cni-k8s#1338

@cp38510 cp38510 closed this as completed Jul 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cp38510