containerd's cli ctr is not pointing at /run/dockershim.sock by default

[benmccown]

What happened: I switched my container runtime to containerd for testing purposes and then connected to an EKS Worker Node's shell to debug a pod I was having issues with. I tried to use ctr to examine my running containers and images, but received an error trying to reach out to the default socket address /run/containerd/containerd.sock

$ ctr version
Client:
  Version:  1.4.6
  Revision: d71fcd7d8303cbf684402823e425e9dd2e99285d
  Go version: go1.15.12

ctr: failed to dial "/run/containerd/containerd.sock": context deadline exceeded

I found /etc/containerd/config.toml pretty quickly and was able to identify that we aren't using the default /run/containerd/containerd.sock but /run/dockershim.sock and ctr has the --address global flag (shown below) where I could specify the right address in my commands, or set the CONTAINERD_ADDRESS variable.

$ ctr help
...
--address value, -a value    address for containerd's GRPC server (default: "/run/containerd/containerd.sock") [$CONTAINERD_ADDRESS]
...

# Resolved with below variable assignment
$ export CONTAINERD_ADDRESS=/run/dockershim.sock

I understand based on #699 that we are intentionally listening on the dockershim socket so that we don't need to change CNI configuration based on the container runtime.

As such I would propose CONTAINERD_ADDRESS be defined for ec2-user/root by bootstrap.sh (set to /run/dockershim.sock) so that ctr will communicate with containerd by default without needing to set the address yourself. I expect a lot of users will be new to containerd and connecting to an instance to look at containers/images only to find yourself communicating with a broken socket isn't a great first experience.

I'm happy enough to submit a PR to set this variable for ec2-user and root via their ~/.bashrc (or maybe globally via /etc/profile?) if we think that is a reasonable solution. The end goal should be that a new containerd user shouldn't need to debug their CLI as their first experience when connecting to a node.

How to reproduce it (as minimally and precisely as possible):

Create a node group that utilizes the containerd runtime (bootstrap --container-runtime flag). Connect to the node via SSH/SSM and then try to run ctr commands.

$ ctr version
$ ctr namespaces ls

Anything else we need to know?:

Environment:

• AWS Region: us-west-2
• Instance Type(s): N/A
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.1
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.21
• AMI Version: ami-068b58ddc0b6e5399 (1.21 for gpu/accelerated workloads)
• Kernel (e.g. uname -a):

$ uname -a
Linux ip-192-168-66-233.us-west-2.compute.internal 5.4.129-62.227.amzn2.x86_64 #1 SMP Wed Jul 7 00:08:43 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

• Release information (run cat /etc/eks/release on a node):

$ cat /etc/eks/release                                                                                                                                                                                                                                      BASE_AMI_ID="ami-01e809b6b105324b3"
BUILD_TIME="Tue Jul 20 01:26:37 UTC 2021"
BUILD_KERNEL="5.4.129-62.227.amzn2.x86_64"
ARCH="x86_64"

[jplock]

Would this be fixed by #724?

[benmccown]

@jplock yes, this should be resolved if #724 is implemented.

[ravisinha0506]

#724 is merged now. Closing this issue as the latest ami ( post v20211013 ) uses containerd default socket path.