Keylime easily deployable on Kubernetes/Openshift.
NOTE: This project is a work in progress.
- Running
Kubernetes
cluster (evidently) - If one wishes to deploy
agents
(which are deployed by default) the following additional requirements should be in place:- All workers nodes should have TPM devices (e.g.,
ls /dev/tpm*
should exit with 0) - For unprivileged pods, K8s version
1.26
or later is required and k8s-tpm-device-plugin shall be installed beforehand. - IMPORTANT: the default mode of operation for this helm chart is to deploy containers with a DameonSet with privileged pods.
- All workers nodes should have TPM devices (e.g.,
a) git clone https://github.com/keylime/attestation-operator.git; cd attestation-operator
b) make help
Usage:
make <target>
General
help Display this help.
Development
manifests Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
generate Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
fmt Run go fmt against code.
vet Run go vet against code.
test Run tests.
build Build manager binary.
run Run a controller from your host.
Development Deployment
install Install CRDs into the K8s cluster specified in ~/.kube/config.
uninstall Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
deploy Deploy controller to the K8s cluster specified in ~/.kube/config.
undeploy Undeploy controller from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
Development Build Dependencies
install-dependencies Downloads and installs all dependencies to LOCALBIN
clean-dependencies Removes all downloaded dependencies from LOCALBIN
kustomize Download kustomize locally if necessary. If wrong version is installed, it will be removed before downloading.
controller-gen Download controller-gen locally if necessary. If wrong version is installed, it will be overwritten.
envtest Download envtest-setup locally if necessary.
helmify Download helmify locally if necessary.
Build
docker-build Builds the application in a docker container and creates a docker image
docker-push Pushes a previously built docker container
helm Builds all helm charts
helm-clean Cleans all packaged helm charts
helm-keylime Builds the keylime helm chart
helm-keylime-clean Cleans the packaged keylime helm chart
helm-keylime-undeploy Undeploy the keylime helm chart
helm-keylime-deploy Deploy the keylime helm chart
helm-keylime-update Update the deployed keylime helm chart
helm-keylime-debug Attempt to debug the keylime helm chart, without deploying it
helm-keylime-push Builds AND pushes the keylime helm chart
helm-keylime-test Basic testing for the keylime helm chart
helm-crds Builds the keylime-crds helm chart
helm-crds-clean Cleans the packaged keylime-crds helm chart
helm-crds-push Builds AND pushes the keylime-crds helm chart
helm-controller Builds the keylime-controller helm chart
helm-controller-push Builds AND pushes the keylime-controller helm chart
c) make helm-keylime
d) make helm-deploy
will deploy an initial barebones (but functional) deployment with 1 registrar
(a Deployment
with a single pod), 1 verifier
(a Deployment
with. a single pod), each backed by their own private sqlite
(in-pod) database and agents
on every node (as a DaemonSet
)
e) make helm-undeploy
will remove the whole deployment
Operator can be deployed with operator-sdk, as it has olm/bundle support.
To do so, you will need operator-sdk
to be installed.
Follow next instructions to compile, upload and deploy through operator-sdk:
- Compile controller, generate bundle and push it to registry:
make docker-build docker-push DOCKER_TAG=quay.io/{quay_user}/attestation-operator:v0.1.0
make bundle DOCKER_TAG=quay.io/{quay_user_here}/attestation-operator:v0.1.0
make bundle-build bundle-push BUNDLE_IMG="quay.io/{quay_user}/attestation-operator-bundle:v0.1.0"
It is possible to generate controller and bundle through podman
. To do so, previous steps must be
replaced with appropriate podman rules:
make podman-build podman-push DOCKER_TAG=quay.io/{quay_user}/attestation-operator:v0.1.0
make podman-bundle DOCKER_TAG=quay.io/{quay_user_here}/attestation-operator:v0.1.0
make podman-bundle-build podman-bundle-push BUNDLE_IMG="quay.io/{quay_user}/attestation-operator-bundle:v0.1.0"
- Deploy through
operator-sdk
tool:
operator-sdk run bundle quay.io/{quay_user}/attestation-operator-bundle:v0.1.0
- In case operator needs to be deployed in a particular namespace, use
--namespace
tool:
operator-sdk run bundle quay.io/{quay_user}/attestation-operator-bundle:v0.1.0 --namespace keylime
- For operator cleanup, just execute
cleanup
operator-sdk option:
operator-sdk cleanup attestation-operator
- In case a namespace was provided for deployment, it must be similarly used on cleanup:
operator-sdk cleanup attestation-operator --namespace keylime
By default, the Makefile
looks for a yaml file on the path set by
the environment variable HELM_CHART_CUSTOM_VALUES
(default
values.yaml
)
This configuration deploys registrar, tenant and verifier pods. The services are made available externally (i.e. the deployment can be used to verify another cluster).
tags:
init: true
registrar: true
verifier: true
agent: false
tenant: true
global:
service:
registrar:
type: NodePort
verifier:
type: NodePort
This configuration deploys a verifier
and registrar
sharing a
MySQL
database (password will be automatically generated and
preserved across (helm
updates)
The mysql server is deployed as part of the helm chart (that is, no extra work is required).
global:
database:
mysql:
enable: true
An external managed MySQL database can be used also for the verifier
and registrar
:
global:
database:
mysql:
external: true
mysql:
auth:
externalIP: "10.103.4.2"
externalUser: "keylime"
externalPassword: "TikBue7mQS"
This configuration demonstrates how to override Keylime configuration parameters. Keylime allows configuration to be overridden by environment variables.
global:
configmap:
configParams:
KEYLIME_AGENT_ENABLE_AGENT_MTLS: 'False'
KEYLIME_AGENT_ENABLE_INSECURE_PAYLOAD: 'True'
KEYLIME_AGENT_RUN_AS: root:root
KEYLIME_AGENT_TPM_ENCRYPTION_ALG: rsa
KEYLIME_AGENT_TPM_HASH_ALG: sha256
KEYLIME_AGENT_TPM_OWNERPASSWORD: temp4now
KEYLIME_AGENT_TPM_SIGNING_ALG: rsassa
KEYLIME_CA_CERT_CA_NAME: feykimeluckers
KEYLIME_CA_CERT_CRL_DIST: http://100.64.255.11:3808/crl
KEYLIME_CA_CERT_LOCALITY: YKT
KEYLIME_CA_CERT_ORGANIZATION: IBM
KEYLIME_CA_CERT_ORG_UNIT: k5l
KEYLIME_CA_CERT_STATE: NY
KEYLIME_CA_PASSWORD: temp4now
KEYLIME_REGISTRAR_AUTO_MIGRATE_DB: 'True'
KEYLIME_REGISTRAR_DATABASE_POOL_SZ_OVFL: 40,80
KEYLIME_TENANT_ENABLE_AGENT_MTLS: 'False'
KEYLIME_TENANT_MAX_PAYLOAD_SIZE: '1048576'
KEYLIME_TENANT_REQUIRE_EK_CERT: 'True'
KEYLIME_VERIFIER_AUTO_MIGRATE_DB: 'True'
KEYLIME_VERIFIER_DATABASE_POOL_SZ_OVFL: 40,80
KEYLIME_VERIFIER_ENABLE_AGENT_MTLS: 'False'
KEYLIME_VERIFIER_QUOTE_INTERVAL: '5'
This configuration deploys Keylime agents
in the cluster in a daemon
set. The pods running the keylime agent
are privileged. We do not
recommend running this configuration in production mode, but may help
with debugging keylime.
tags:
agent: true
global:
service:
agent:
privileged: true
This configuration deploys Keylime agents
in the cluster, but the pods
running it are unprivileged. The keylime agent
needs access to
the TPM device and to parts of the securityfs
file system. Both of
these are provided by the TPM device plugin, which is turned on
automatically with unprivileged agent
pods.
In addition, the effective group ID of the keylime agent
pod has to
match the group ID of the TPM device. Kubernetes does not allow
runAsGroup
to take symbolic values.
TODO instructions for mounting the MBA and IMA logs in nonstandard places.
tags:
agent: true
global:
service:
agent:
privileged: false
keylime-agent:
unprivsecurityContext:
readOnlyRootFilesystem: true
privileged: false
capabilities:
drop:
- ALL
runAsGroup: 109 <---- make this match the group ID of group <tss> on the hosts running the agent.
This configuration is for those of us debugging custom (self-built) keylime images. We imagine this to be the standard way to do Keylime development.
global:
service:
agent:
initImage:
repository: localhost/custom-agent-initImage
tag: latest
image:
repository: localhost/custom-agent-image
tag: latest
registrar:
image:
repository: localhost/custom-registrar-image
tag: latest
verifier:
image:
repository: localhost/custom-verifier-image
tag: latest
tenant:
image:
repository: localhost/custom-tenant-image
tag: latest
For both availability and scalability reasons (scale-out), multiple verifiers
and registrars
can be deployed. Please do notice that this configuration
cannot operate with a sqlite
backend (i.e. either mysql:external
or
mysql:enable
has to be set to true
). When adding a new agent
to a
verifier
, one of the multiple ones (exposed via multiple services, one per
pod in the StatefulSet
) has to be selected. The utility kt
, used by make helm-keylime-test
shows an example on how to pick a verifier
based on an
agent
UUID.
global:
database:
mysql:
enable: true
service:
registrar:
replicas: 2
type: "LoadBalancer"
verifier:
replicas: 3
type: "LoadBalancer"
As a convenience during development, each individual service can be set to
"developer mode", resulting in a pod
with a long sleep time (7 days or
604,800 seconds). The developer can then access the pod
(via kubectl exec -i
) and perform multiple iterations of code change followed by service
(re)start (e.g., keylime_verifier
)
global:
service:
registrar:
developer: false
verifier:
developer: true
agent:
developer: true