can't execute 'ctr': No such file or directory

[bugslifesolutions]

containerd://1.6.8

kubectl sniff rabbitmq-server-0 -p -n applianceshack -v
INFO[0000] running in verbose mode                      
DEBU[0000] pod 'rabbitmq-server-0' status: 'Running'    
INFO[0000] no container specified, taking first container we found in pod. 
INFO[0000] selected container: 'rabbitmq'               
INFO[0000] sniffing method: privileged pod              
INFO[0000] sniffing on pod: 'rabbitmq-server-0' [namespace: 'applianceshack', container: 'rabbitmq', filter: '', interface: 'any'] 
INFO[0000] creating privileged pod on node: 'worker-1'  
DEBU[0000] creating privileged pod on remote node       
W1201 02:54:40.768551  191132 warnings.go:70] would violate PodSecurity "restricted:latest": host namespaces (hostPID=true), privileged (container "ksniff-privileged" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "ksniff-privileged" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "ksniff-privileged" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volumes "host", "container-socket" use restricted volume type "hostPath"), runAsNonRoot != true (pod or container "ksniff-privileged" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "ksniff-privileged" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
INFO[0000] pod: 'ksniff-zcl7z' created successfully in namespace: 'applianceshack' 
DEBU[0000] created pod details: &Pod{ObjectMeta:{ksniff-zcl7z ksniff- applianceshack  578d9a20-1987-4521-b38e-7da90d3da8c2 14886321 0 2022-12-01 02:54:40 +0000 UTC <nil> <nil> map[app:ksniff] map[] [] []  [{kubectl-sniff Update v1 2022-12-01 02:54:40 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"ksniff-privileged\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:securityContext":{".":{},"f:privileged":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/host\"}":{".":{},"f:mountPath":{},"f:name":{}},"k:{\"mountPath\":\"/run/containerd/containerd.sock\"}":{".":{},"f:mountPath":{},"f:name":{},"f:readOnly":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:hostPID":{},"f:nodeName":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"container-socket\"}":{".":{},"f:hostPath":{".":{},"f:path":{},"f:type":{}},"f:name":{}},"k:{\"name\":\"host\"}":{".":{},"f:hostPath":{".":{},"f:path":{},"f:type":{}},"f:name":{}}}}}}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:host,VolumeSource:VolumeSource{HostPath:&HostPathVolumeSource{Path:/,Type:*Directory,},EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,Ephemeral:nil,},},Volume{Name:container-socket,VolumeSource:VolumeSource{HostPath:&HostPathVolumeSource{Path:/run/containerd/containerd.sock,Type:*Socket,},EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,Ephemeral:nil,},},Volume{Name:kube-api-access-4ccqt,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:ksniff-privileged,Image:docker.io/hamravesh/ksniff-helper:v3,Command:[sh -c sleep 10000000],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:container-socket,ReadOnly:true,MountPath:/run/containerd/containerd.sock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:host,ReadOnly:false,MountPath:/host,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-4ccqt,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:worker-1,HostNetwork:false,HostPID:true,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{},Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[]ContainerStatus{},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},} 
INFO[0000] waiting for pod successful startup           
INFO[0002] pod: 'ksniff-zcl7z' created successfully on node: 'worker-1' 
INFO[0002] spawning wireshark!                          
INFO[0002] starting remote sniffing using privileged pod 
INFO[0002] executing command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
    export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
    crictl pull docker.io/maintained/tcpdump:latest >/dev/null
    netns=$(crictl inspect 3d5503facffd72c58406eb812425f387e72c8b4a8fdcd8e72e0c44f9aac08b87 | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
    exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-VDipuLFu tcpdump -i any -U -w -  
    ]' on container: 'ksniff-privileged', pod: 'ksniff-zcl7z', namespace: 'applianceshack' 
INFO[0002] command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
    export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
    crictl pull docker.io/maintained/tcpdump:latest >/dev/null
    netns=$(crictl inspect 3d5503facffd72c58406eb812425f387e72c8b4a8fdcd8e72e0c44f9aac08b87 | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
    exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-VDipuLFu tcpdump -i any -U -w -  
    ]' executing successfully exitCode: '127', stdErr :'+ export 'CONTAINERD_SOCKET=/run/containerd/containerd.sock'
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_RUNTIME_ENDPOINT=unix:///host/run/containerd/containerd.sock'
+ export 'IMAGE_SERVICE_ENDPOINT=unix:///host/run/containerd/containerd.sock'
+ crictl pull docker.io/maintained/tcpdump:latest
+ crictl inspect 3d5503facffd72c58406eb812425f387e72c8b4a8fdcd8e72e0c44f9aac08b87
+ jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path'
+ tr -d '"'
+ netns=/proc/178883/ns/net
+ exec chroot /host ctr -a /run/containerd/containerd.sock run --rm --with-ns network:/proc/178883/ns/net docker.io/maintained/tcpdump:latest ksniff-container-VDipuLFu tcpdump -i any -U -w -
chroot: can't execute 'ctr': No such file or directory
' 
INFO[0002] remote sniffing using privileged pod completed 
INFO[0003] starting sniffer cleanup                     
INFO[0003] removing privileged container: 'ksniff-privileged' 
INFO[0003] executing command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_ID="ksniff-container-VDipuLFu"
    chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
    ]' on container: 'ksniff-privileged', pod: 'ksniff-zcl7z', namespace: 'applianceshack' 
INFO[0003] command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_ID="ksniff-container-VDipuLFu"
    chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
    ]' executing successfully exitCode: '127', stdErr :'+ export 'CONTAINERD_SOCKET=/run/containerd/containerd.sock'
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_ID=ksniff-container-VDipuLFu'
+ chroot /host ctr -a /run/containerd/containerd.sock task kill -s SIGKILL ksniff-container-VDipuLFu
chroot: can't execute 'ctr': No such file or directory
' 
INFO[0003] privileged container: 'ksniff-privileged' removed successfully 
INFO[0003] removing pod: 'ksniff-zcl7z'                 
INFO[0003] removing privileged pod: 'ksniff-zcl7z'      
INFO[0003] privileged pod: 'ksniff-zcl7z' removed       
INFO[0003] pod: 'ksniff-zcl7z' removed successfully     
INFO[0003] sniffer cleanup completed successfully       
Error: signal: aborted (core dumped)

k get nodes -o wide
NAME             STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
controlplane-1   Ready    control-plane   29d    v1.25.2   10.188.0.11   <none>        Talos (v1.2.5)   5.15.72-talos    containerd://1.6.8
controlplane-2   Ready    control-plane   28d    v1.25.2   10.188.0.14   <none>        Talos (v1.2.5)   5.15.72-talos    containerd://1.6.8
controlplane-3   Ready    control-plane   25d    v1.25.2   10.188.0.15   <none>        Talos (v1.2.5)   5.15.72-talos    containerd://1.6.8
worker-1         Ready    <none>          25d    v1.25.2   10.188.0.16   <none>        Talos (v1.2.5)   5.15.72-talos    containerd://1.6.8
worker-2         Ready    <none>          25d    v1.25.2   10.188.0.17   <none>        Talos (v1.2.5)   5.15.72-talos    containerd://1.6.8
worker-3         Ready    <none>          24d    v1.25.2   10.188.0.18   <none>        Talos (v1.2.5)   5.15.72-talos    containerd://1.6.8
worker-4         Ready    <none>          2d9h   v1.25.2   10.188.0.21   <none>        Talos (v1.2.5)   5.15.72-talos    containerd://1.6.8

[isbalashov]

Workaround for containerd installation: just remove -p and it should work
Like this kubectl sniff rabbitmq-server-0 -n applianceshack -v

[alphabet5]

Without -p I get this error

INFO[0003] command: '[/tmp/static-tcpdump -i any -U -w - ]' executing successfully exitCode: '1', stdErr :'static-tcpdump: any: You don't have permission to capture on that device
(socket: Operation not permitted)
'
ERRO[0003] failed to start remote sniffing, stopping wireshark  error="executing sniffer failed, exit code: '1'"

[diafour]

In my case ctr was not in the default path. I've create temporary symlink on node cd /bin ; ln -s /path/to/ctr ctr and it works.

[netthier]

What should be done in the case of nodes with read-only file systems, like Talos Linux?