Incompatibility WorkloadAttestation k8s and Kubeedge k8s

[goergch]

We have a kubernetes cluster, which includes nodes, that are connected via kubeedge (https://kubeedge.io/) and run cri-o. In this system we are using spire and also the kubeedge nodes run spire agents.

Sadly the workloadattestor k8s was incompatible with our system, but we managed to patch it temporarily with spire v1.0.0.
The problem is that /proc/{PID}/cgroup looks different on our system, than on a "normal" k8s node.

"Normal" k8s node:

12:hugetlb:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
11:freezer:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
10:net_cls,net_prio:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
9:perf_event:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
8:memory:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
7:blkio:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
6:pids:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
5:cpu,cpuacct:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
4:rdma:/
3:cpuset:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
2:devices:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
1:name=systemd:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169fbc47_41d0_4df5_be62_6b231010b6c4.slice/docker-bbf76635888294204963d9e32e4fae57dcbf8eddb7adeecbd55672cd5fbc2249.scope
0::/

our edge node:

0::/../crio-45490e76e0878aaa4d9808f7d2eefba37f093c3efbba9838b6d8ab804d9bd814.scope

As said, patching the workloadattestor in spire 1.0.0 was possible by adapting the regex extracting the container id.
But since #2600 v1.1.1, the algorithm also checks for PodUID. As this information is not available in the cgroup of our system, patching is no more possible.

A solution inside spire is highly appreciated. But hints on how to configure our edge node could also help.

• Version: v1.2.3
• Platform: Linux 1bdf700d-f0dd-4eb4-89eb-47eb8acfd704 5.18.4-1-default CLI authentication #1 SMP PREEMPT_DYNAMIC Wed Jun 15 06:00:33 UTC 2022 (ed6345d) x86_64 x86_64 x86_64 GNU/Linux, MicroOS Opensuse, cri-o, kubeedge
• Subsystem: workloadattestor k8s

[azdagron]

Hmm, that's interesting. I don't know much experience with either crio or kubeedge. If there were a way to influence cgroup structure that would be ideal.

However, that being said, the extraction of the pod uid is currently to short-circuit searching through unrelated pods for the container running the workload process. I think we could probably support this kind of setup by doing the following:

1. Allowing the configured regex to only select the container id by choosing whether the regex had one or two match groups. If two, assume that match group 1 is the pod id and match group 2 is the container id (today's behavior). If only a single match group, assume it is the container id.
2. If the pod id is not extracted, then search all pods in the listing for the target container. If more than one pod is identified as having that container (which would be an exceptional circumstance and likely a bug) then return error from workload attestation.

[goergch]

Issue is fixed by #3242
The issue is rather kubeedge than cri-o,