Skip to content

Commit

Permalink
New container locator for docker/k8s on linux (#5076)
Browse files Browse the repository at this point in the history 
* New container locator for docker/k8s on linux

The docker and k8s workload attestors work backwards from pid to
container by inspecting the proc filesystem. Today, this happens by
inspecting the cgroup file. Identifying the container ID (and pod UID)
from the cgroup file has been a continual arms race. The k8s and docker
workload attestors grew different mechanisms for trying to deal with the
large variety in the output.

Further, with cgroups v2 and private namespaces, the cgroup file might
not have the container ID or pod UID information within it.

This PR unifies the container ID (and pod UID) extraction for both the
docker and k8s workload attestors. The new implementation searches the
mountinfo file first for cgroups mounts. If not found, it will fall back
to the cgroup file (typically necessary only when the workload is
running in the same container as the agent).

The extraction algorithm is the same for both mountinfo and cgroup
entries, and is as follows:
1. Iterator over each entry in the file being searched, extracting
   either the cgroup mount root (mountinfo) or the cgroup group
   path (cgroup) as the source path.
2. Walk backwards through the segments in the source path looking for
   the 64-bit hex digit container ID.
3. If looking for the pod UID (K8s only), then walk backwards through
   the segments in the path looking for the pod UID pattern used by
   kubelet. Start with the segment the container ID was found in
   (truncated to remove the container ID portion).
4. If there are pod UID/container ID conflicts after searching these
   files then log and abort. Entries that have a pod UID override those
   that don't.

The container ID is very often contained in the last segment in the path
but there are situations where it isn't.

This new functionality is NOT enabled by default, but opted in using the
`use_new_container_locator` configurable in each plugin. In 1.10, we can
consider enabling it by default.

The testing for the new code is spread out a little bit. The cgroups
fallback functionality is mostly tested by the existing tests in the
k8s and docker plugin tests. The mountinfo tests are only in the new
containerinfo package.

In the long term, I'd like to see all of the container info extraction
related tests moved solely to the containerinfo package and removed from
the individual plugins.

Resolves #4004, resolves #4682, resolves #4917.

Signed-off-by: Andrew Harding <azdagron@gmail.com>

* missing new arg

Signed-off-by: Andrew Harding <azdagron@gmail.com>

* fix windows tests

Signed-off-by: Andrew Harding <azdagron@gmail.com>

* fix windows tests and lint

Signed-off-by: Andrew Harding <azdagron@gmail.com>

* address pr comments

Signed-off-by: Andrew Harding <azdagron@gmail.com>

* markdown lint

Signed-off-by: Andrew Harding <azdagron@gmail.com>

* add agent full conf

Signed-off-by: Andrew Harding <azdagron@gmail.com>

* fix labels

Signed-off-by: Andrew Harding <azdagron@gmail.com>

* change log to warn

Signed-off-by: Andrew Harding <azdagron@gmail.com>

* use new locator in it

Signed-off-by: Andrew Harding <azdagron@gmail.com>

---------

Signed-off-by: Andrew Harding <azdagron@gmail.com>
  • Loading branch information
@azdagron
azdagron authored Apr 24, 2024
1 parent 6760216 commit 8090bf3
Show file tree
Hide file tree
Showing 31 changed files with 702 additions and 125 deletions.
36 changes: 28 additions & 8 deletions conf/agent/agent_full.conf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -90,15 +90,15 @@ agent {
# # default X.509 bundle with Envoy SDS. Default: ROOTCA.
# # default_bundle_name = "ROOTCA"
#
# # default_all_bundles_name: The Validation Context resource name to use to fetch
# # default_all_bundles_name: The Validation Context resource name to use to fetch
# # all bundles (including federated bundles) with Envoy SDS. Cannot be used with
# # Envoy releases prior to 1.18.
# # default_all_bundles_name = "ALL"

# # disable_spiffe_cert_validation: disable Envoy SDS custom SPIFFE validation. Default: false
# # disable_spiffe_cert_validation = false
# }

# allowed_foreign_jwt_claims: set a list of trusted claims to be returned when validating foreign JWTSVIDs
# allowed_foreign_jwt_claims = []

Expand Down Expand Up @@ -259,7 +259,7 @@ plugins {
NodeAttestor "tpm_devid" {
plugin_data {
# tpm_device_path: Optional. The path to a TPM 2.0 device. If unset
# the plugin will try to autodetect the TPM path. It is not used when running
# the plugin will try to autodetect the TPM path. It is not used when running
# on windows.
# tpm_device_path = "/dev/tpmrm0"

Expand All @@ -286,7 +286,7 @@ plugins {
# devid_password = "password"
}
}

# SVIDStore "gcp_secretmanager": An SVID store that stores the SVIDs in
# Google Cloud Secret Manager.
SVIDStore "gcp_secretmanager" {
Expand All @@ -310,7 +310,7 @@ plugins {
# secret_access_key = ""

# region: AWS region to store the secrets.
# region = ""
# region = ""
}
}

Expand All @@ -324,6 +324,16 @@ plugins {
# docker_version: The API version of the docker daemon. If not
# specified, the version is negotiated by the client.
# docker_version = ""

# use_new_container_locator: If true, enables the new container
# locator algorithm that has support for cgroups v2. Default:
# false. (Linux only)
# use_new_container_locator = false

# verbose_container_locator_logs: If true, enables verbose logging
# of mountinfo and cgroup information used to locate containers.
# Defaults to false. (Linux only)
# verbose_container_locator_logs = false
}
}

Expand Down Expand Up @@ -360,7 +370,7 @@ plugins {
# private_key_path: The path on disk to client key used for kubelet
# authentication.
# private_key_path = ""

# use_anonymous_authentication: If true, use anonymous authentication
# for kubelet communication.
# use_anonymous_authentication = false
Expand All @@ -373,13 +383,23 @@ plugins {
# the environment variable specified by node_name_env.
# node_name = ""

# use_new_container_locator: If true, enables the new container
# locator algorithm that has support for cgroups v2. Default:
# false. (Linux only)
# use_new_container_locator = false

# verbose_container_locator_logs: If true, enables verbose logging
# of mountinfo and cgroup information used to locate containers.
# Defaults to false. (Linux only)
# verbose_container_locator_logs = false

# experimental: Experimental features.
experimental {
# sigstore: sigstore options. Enables signature checking.
# sigstore {
# rekor_url: The URL for the rekor STL Server to use with cosign. Required.
# rekor_url = "https://rekor.sigstore.dev"

# skip_signature_verification_image_list: List of images that should
# not be verified by cosign. They will receive a default
# sigstore-validation:passed selector, but no other sigstore related selectors.
Expand Down
14 changes: 8 additions & 6 deletions doc/plugin_agent_workloadattestor_docker.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,14 @@ The `docker` plugin generates selectors based on docker labels for workloads cal
It does so by retrieving the workload's container ID from its cgroup membership on Unix systems or Job Object names on Windows,
then querying the docker daemon for the container's labels.

| Configuration | Description | Default |
|------------------------------|------------------------------------------------------------------------------|----------------------------------|
| docker_socket_path | The location of the docker daemon socket (Unix) | "unix:///var/run/docker.sock" |
| docker_version | The API version of the docker daemon. If not specified | |
| container_id_cgroup_matchers | A list of patterns used to discover container IDs from cgroup entries (Unix) |
| docker_host | The location of the Docker Engine API endpoint (Windows only) | "npipe:////./pipe/docker_engine" |
| Configuration | Description | Default |
|--------------------------------|------------------------------------------------------------------------------------------------|----------------------------------|
| docker_socket_path | The location of the docker daemon socket (Unix) | "unix:///var/run/docker.sock" |
| docker_version | The API version of the docker daemon. If not specified | |
| container_id_cgroup_matchers | A list of patterns used to discover container IDs from cgroup entries (Unix) | |
| docker_host | The location of the Docker Engine API endpoint (Windows only) | "npipe:////./pipe/docker_engine" |
| use_new_container_locator | If true, enables the new container locator algorithm that has support for cgroups v2 | false |
| verbose_container_locator_logs | If true, enables verbose logging of mountinfo and cgroup information used to locate containers | false |

A sample configuration:

Expand Down
30 changes: 16 additions & 14 deletions doc/plugin_agent_workloadattestor_k8s.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,20 +46,22 @@ server name validation against the kubelet certificate.
**Note** To run on Windows containers, Kubernetes v1.24+ and containerd v1.6+ are required,
since [hostprocess](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/) container is required on the agent container.

| Configuration | Description |
|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `disable_container_selectors` | If true, container selectors are not produced. This can be used to produce pod selectors when the workload pod is known but the workload container is not ready at the time of attestation. |
| `kubelet_read_only_port` | The kubelet read-only port. This is mutually exclusive with `kubelet_secure_port`. |
| `kubelet_secure_port` | The kubelet secure port. It defaults to `10250` unless `kubelet_read_only_port` is set. |
| `kubelet_ca_path` | The path on disk to a file containing CA certificates used to verify the kubelet certificate. Required unless `skip_kubelet_verification` is set. Defaults to the cluster CA bundle `/run/secrets/kubernetes.io/serviceaccount/ca.crt`. |
| `skip_kubelet_verification` | If true, kubelet certificate verification is skipped |
| `token_path` | The path on disk to the bearer token used for kubelet authentication. Defaults to the service account token `/run/secrets/kubernetes.io/serviceaccount/token` |
| `certificate_path` | The path on disk to client certificate used for kubelet authentication |
| `private_key_path` | The path on disk to client key used for kubelet authentication |
| `use_anonymous_authentication` | If true, use anonymous authentication for kubelet communication |
| `node_name_env` | The environment variable used to obtain the node name. Defaults to `MY_NODE_NAME`. |
| `node_name` | The name of the node. Overrides the value obtained by the environment variable specified by `node_name_env`. |
| `experimental` | The experimental options that are subject to change or removal. |
| Configuration | Description |
|-------------------------------- |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `disable_container_selectors` | If true, container selectors are not produced. This can be used to produce pod selectors when the workload pod is known but the workload container is not ready at the time of attestation. |
| `kubelet_read_only_port` | The kubelet read-only port. This is mutually exclusive with `kubelet_secure_port`. |
| `kubelet_secure_port` | The kubelet secure port. It defaults to `10250` unless `kubelet_read_only_port` is set. |
| `kubelet_ca_path` | The path on disk to a file containing CA certificates used to verify the kubelet certificate. Required unless `skip_kubelet_verification` is set. Defaults to the cluster CA bundle `/run/secrets/kubernetes.io/serviceaccount/ca.crt`. |
| `skip_kubelet_verification` | If true, kubelet certificate verification is skipped |
| `token_path` | The path on disk to the bearer token used for kubelet authentication. Defaults to the service account token `/run/secrets/kubernetes.io/serviceaccount/token` |
| `certificate_path` | The path on disk to client certificate used for kubelet authentication |
| `private_key_path` | The path on disk to client key used for kubelet authentication |
| `use_anonymous_authentication` | If true, use anonymous authentication for kubelet communication |
| `node_name_env` | The environment variable used to obtain the node name. Defaults to `MY_NODE_NAME`. |
| `node_name` | The name of the node. Overrides the value obtained by the environment variable specified by `node_name_env`. |
| `experimental` | The experimental options that are subject to change or removal. |
| `use_new_container_locator` | If true, enables the new container locator algorithm that has support for cgroups v2. Defaults to false. |
| `verbose_container_locator_logs` | If true, enables verbose logging of mountinfo and cgroup information used to locate containers. Defaults to false. |

| Experimental options | Description |
|----------------------|----------------------------------------------------------------------------------------------------------------------------- |
Expand Down
4 changes: 4 additions & 0 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ require (
github.com/go-sql-driver/mysql v1.8.1
github.com/godbus/dbus/v5 v5.1.0
github.com/gofrs/uuid/v5 v5.1.0
github.com/gogo/status v1.1.1
github.com/google/btree v1.1.2
github.com/google/go-cmp v0.6.0
github.com/google/go-containerregistry v0.19.1
Expand Down Expand Up @@ -89,6 +90,7 @@ require (
k8s.io/apimachinery v0.30.0
k8s.io/client-go v0.30.0
k8s.io/kube-aggregator v0.30.0
k8s.io/mount-utils v0.30.0
sigs.k8s.io/controller-runtime v0.17.3
)

Expand Down Expand Up @@ -197,6 +199,7 @@ require (
github.com/go-openapi/validate v0.24.0 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/gogo/googleapis v0.0.0-20180223154316-0cd9801be74a // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
github.com/golang-jwt/jwt/v5 v5.2.1 // indirect
Expand Down Expand Up @@ -259,6 +262,7 @@ require (
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/sys/mountinfo v0.6.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mozillazg/docker-credential-acr-helper v0.3.0 // indirect
Expand Down
Loading

0 comments on commit 8090bf3

Please sign in to comment.