Enhance the usability of the Libvirt provider for confidential container administrators

[savitrilh]

Improve the usability of the Libvirt provider for confidential container administrators

Summary

To improve the user experience for confidential container administrators to enable the confidential container functionality, sandbox container operators should utilize a specialized script designed for Libvirt provider for managing administrative tasks specific to confidential containers during the activation of Peerpod through the KataConfig configmap.

Motivation

Currently, the OpenShift Sandbox Container(OSC) Operator manages the following tasks:

• This manages the lifecycle (install/configure/update) of sandboxed containers runtime (Kata containers) on OpenShift clusters.
• Build the PODVM QCOW2 image or pull the PODVM image from the registry, extract the PODVM image from a container wrapped QCOW2 image.
• Uploads the operator built or pulled pre-built PODVM image to pre-created Libvirt volume.

OSC operator handles above operations automatically but still it requires following manual operations to be executed by confidential container administrator:

1. Generation of SSH Key(public-private key) in OCP cluster.

$ ssh-keygen -f ./id_rsa -N ""

2. Copy the SSH public key to KVM Host(LPAR). So that worker node can perform Libvirt specific operations such as creating the peerpod on the LPAR(KVM host) and lifecycle management of the peerpod VM.

$ ssh-copy-id -i ./id_rsa.pub <KVM_HOST_IP_ADDRESS>

3. Create the SSH secret object

$ oc create secret generic ssh-key-secret \
    -n openshift-sandboxed-containers-operator \
    --from-file=id_rsa.pub=./id_rsa.pub \
    --from-file=id_rsa=./id_rsa

4. Create Libvirt POOL and Volume

Set the name of the libvirt pool by running the following command:
$ export LIBVIRT_POOL=<name_of_libvirt_pool_to_create>

Set the name of the libvirt volume by running the following command:
$ export LIBVIRT_VOL_NAME=<name_of_libvirt_volume_to_create>

Set the path of the default storage pool location, by running the following command:
$ export LIBVIRT_POOL_DIRECTORY=<name_of_target_directory>

Note:To ensure libvirt has read and write access permissions, use a subdirectory of libvirt’s storage directory. The default is /var/lib/libvirt/images/

Create a libvirt pool by running the following command:
$ virsh pool-define-as $LIBVIRT_POOL --type dir --target "$LIBVIRT_POOL_DIRECTORY"

Start the libvirt pool by running the following command:
$ virsh pool-start $LIBVIRT_POOL

5. Create and apply the PeerPod Secret and ConfigMap.

1. Create and apply the PeerPod Secret

Create a YAML file peer-pods-secret.yaml with the following manifest:

apiVersion: v1
kind: Secret
metadata:
  name: peer-pods-secret
  namespace: openshift-sandboxed-containers-operator
type: Opaque
stringData:
  CLOUD_PROVIDER: "libvirt" [1](https://docs.redhat.com/en/documentation/openshift_sandboxed_containers/1.5/html-single/openshift_sandboxed_containers_user_guide/index#CO17-1)
  LIBVIRT_URI: "<libvirt_gateway_uri>" [2](https://docs.redhat.com/en/documentation/openshift_sandboxed_containers/1.5/html-single/openshift_sandboxed_containers_user_guide/index#CO17-2)
  LIBVIRT_POOL: "<libvirt_pool>" 3
  LIBVIRT_VOL_NAME: "<libvirt_volume>" 4

1 Enter libvirt as the cloud provider.
2 Enter the libvirt_gateway_uri value you retrieved.
3 (https://docs.redhat.com/en/documentation/openshift_sandboxed_containers/1.5/html-single/openshift_sandboxed_containers_user_guide/index#CO17-3) Enter the libvirt_pool value you retrieved.
4 (https://docs.redhat.com/en/documentation/openshift_sandboxed_containers/1.5/html-single/openshift_sandboxed_containers_user_guide/index#CO17-4) Enter the libvirt_volume value you retrieved.

2. Create the peer-pod-secret object:

$ oc apply -f peer-pods-secret.yaml

3. Create and apply ConfigMap.

Create a peerpod configuration file(peer-pods-cm.yaml)

apiVersion: v1
kind: ConfigMap
metadata:
  name: peer-pods-cm
  namespace: openshift-sandboxed-containers-operator
data:
  CLOUD_PROVIDER: "libvirt"
  PROXY_TIMEOUT: "15m"

Apply the ConfigMap:

$ oc apply -f peer-pods-cm.yaml

4. Create Libvirt image config map
i. For Operator built PodVM image

apiVersion: v1
kind: ConfigMap
metadata:
  name: libvirt-podvm-image-cm
  namespace: openshift-sandboxed-containers-operator
data:
  # PodVM image distro
  PODVM_DISTRO: "rhel"

  # Pod VM sources
  # If changing the source, then ensure the respective payload binaries are available
  # for the new source
  CAA_SRC: "https://github.com/confidential-containers/cloud-api-adaptor"
  CAA_REF: "v0.8.2"

  # Booleans
  DOWNLOAD_SOURCES: "no"
  CONFIDENTIAL_COMPUTE_ENABLED: "no"
  UPDATE_PEERPODS_CM: "yes"

  # Libvirt specific
  ORG_ID: ""
  ACTIVATION_KEY: ""
  BASE_OS_VERSION: "9.4"

  # To Enable SE for IBM Z
  SE_BOOT: "true"

ii. For Pre built PodVM image

apiVersion: v1
kind: ConfigMap
metadata:
  name: libvirt-podvm-image-cm
  namespace: openshift-sandboxed-containers-operator
data:
  # PodVM image distro
  PODVM_DISTRO: "rhel"

  # Pod VM sources
  # If changing the source, then ensure the respective payload binaries are available
  # for the new source
  CAA_SRC: "https://github.com/confidential-containers/cloud-api-adaptor"
  CAA_REF: "v0.8.2"

  # Booleans
  DOWNLOAD_SOURCES: "no"
  CONFIDENTIAL_COMPUTE_ENABLED: "no"
  UPDATE_PEERPODS_CM: "yes"

  # Libvirt specific
  ORG_ID: ""
  ACTIVATION_KEY: ""
  BASE_OS_VERSION: "9.4

  # For Pre-built PodVM images.
  PODVM_IMAGE_URI: "" # eg: oci::quay.io/openshift_sandboxed_containers/libvirt-podvm-image:latest::/image/podvm.qcow2

The goal is to enhance user experience by automating the aforementioned manual operations through a script. These scripts will be called from OSC through job.

Goals

To improve the user experience for confidential container administrators to enable the confidential container functionality, sandbox container operators should utilize a specialized script designed for Libvirt provider for managing administrative tasks specific to confidential containers during the activation or de-activation of Peerpod through the KataConfig configmap.

Proposal

This new script libvirt-config-manager.sh will be developed and shipped with openshift sandbox container operator. This script can be called for s390x Libvirt provider in the reconcile path(routine) of the openshift sandbox container in the functions processKataConfigInstallRequest and handleFeatureConfidential. This reduces the manual operation.

Design

This design focuses to improve the usability experience for the OCP Admin user to enable the confidential containers.

Four new files libvirt-podvm-secret.yaml, libvirt-config-manager.sh, osc-podvm-pre-config-job.yaml and osc-cleanup.yaml will be added to improve the usability.

libvirt-podvm-secret.yaml file will contain the following fields(Keys)

1. KVM HOST(LPAR) IP -> where OCP worker nodes are running
2. KVM HOST Username
3. KVM HOST password (Encode the password)
4. Directory path (Location of the Libvirt Pool)
5. Libvirt Pool Name
6. Libvirt Volume name
7. SE_BOOT: "true
8. PODVM_IMAGE_URI
   ......
   KVM HOST(LPAR) IP, KVM HOST Username and KVM HOST password are mandatory fields. Libvirt Pool Path, Pool Name Volume names, SE_BOOT and PODVM_IMAGE_URI are optional.

Note: Administrators need to create a user on the LPAR with the necessary permissions to execute virsh commands and apply the Libvirt configuration filelibvirt-podvm-secret.yaml on the OpenShift Container Platform (OCP) before enabling peerpods through KataConfig configmap.

The libvirt-config-manager.sh script utilizes the administrative user's specified data from the libvirt-podvm-secret.yaml file.

The libvirt-config-manager.sh script sequentially executes the following tasks.

1. Validate the mandatory field values in the libvirt-podvm-secret.yaml file.

2. Generate a SSH key pair (public-private key) within the OCP cluster.

3. Decode the encoded KVM host password as specified in libvirt-podvm-secret.yaml.

4. Copy the generated SSH public key to the KVM Host(LPAR).

5. Create the SSH secret object on OCP

6. Create the directory on the KVM host where the user specifies the path for the default storage pool location.

7. Create a Libvirt pool using the specified pool name if specified by the admin user in the libvirt-podvm-secret.yaml file. Otherwise, create the libvirt pool using the default pool name.

8. Start the created libvirt pool

9. Create a Libvirt volume using the specified volume name if specified by the admin user in the libvirt-podvm-secret.yaml file. Otherwise, create the libvirt volume using the default volume name.

10. Generate and apply PeerPod Secret, PeerPod ConfigMap and Libvirt Image ConfigMap.

This script libvirt-config-manager.sh will be shipped with openshift sandbox container operator. This script can be called for s390x Libvirt provider in reconcile path(routine) of the openshift sandbox container in the functions processKataConfigInstallRequest and handleFeatureConfidential through the jobs osc-podvm-pre-config-job.yaml and osc-cleanup.yaml to create Secrets, ConfigMaps, Libvirt Pool and Volume during PeerPod activation and remove them during PeerPod de-activation through jobs.

Future Work Items related to this

Currently, the KVM host (LPAR) password is encoded and applied via script for the Tech Preview phase. For the General Availability (GA) release, TLS server certificates will be generated and stored securely, similar to other credentials. The plan is to establish a TLS connection between the LPAR and OCP host, enabling secure transmission of the SSH public key generated on OCP to the LPAR.

Conclusion

This approach improves user experience for confidential container administrators to enable/disable the confidential container functionality, sandbox container operators should utilize a specialized
script designed for Libvirt provider for managing administrative tasks specific to confidential containers during the activation or de-activation of Peerpod through the KataConfig configmap.

This proposal can be found in following google document:
https://docs.google.com/document/d/1ZfgG2PJXs4FeKeV5RkGGkrJ_MLZJxX5smSGrOiN8ZUw/edit

[bpradipt]

Here is my understanding of the issue:

Before creating a podVM image for libvirt (either operator built or from a pre-built image) an administrator must perform the following steps

1. Create peer-pods-secret object with access details for the libvirt host
2. Create peer-pods-cm object with details on the libvirt volume (pool, name etc) where the podvm image will be stored

After the above steps the KataConfig will be created which will then create the podvm image. And you want to automate step 1 and 2 for a better admin experience.

Is this the correct problem summary and goal ?

[savitrilh]

Yes, @bpradipt . In addition to items 1 and 2, I would like to automate the following:

1. SSH key Pair creation and copying SSH Public Key to LPAR.
2. SSH secret creation.

Additionally, as we discussed, it would be beneficial to decouple the following optional tasks:

1. Libvirt pool and volume creation.
2. Libvirt image file creation.

[bpradipt]

Now its much clearer to me.

At a high level I think the following functionality can be implemented first:

• Add the libvirt pool/volume creation details in the libvirt image handler and perform the required operation before creating the image (similar to how for Azure provider the image gallery is created). This won't require any new job, just changes to the libvirt handler.

Regarding SSH key, afaik all the providers require SSH key ( for non CoCo). Can there be a generic method that applies to all the providers? And does this needs to be in the OSC operator or cloud-api-adaptor ?

As for automating the bootstrap configuration that needs to go in configmap and secrets it needs to be discussed how it can work for all the providers, as the problem is same for all the providers. And does this functionality need to be in the OSC operator or cloud-api-adaptor ?

cc @gkurz @snir911 @littlejawa

[savitrilh]

@bpradipt, the Libvirt volume name information needs to be included in the peer-pod-secret. Therefore, we handled all the creation steps through the same job and handler.

Here are my thoughts:

It’s more efficient to manage SSH key pair generation and the creation of the secret object using OpenShift Sandbox Container Operator(OSC) rather than the cloud-api-adapter because:

• We generate one SSH key pair and create a secret containing that information, making the same secret available to all worker nodes. We then copy the SSH public key to the LPAR.

• The cloud-api-adapter operates as a DaemonSet running on each worker node.

• If we implement the generation logic in the cloud-api-adapter, each instance will create its own SSH key pair.

• This creates a challenge: how can we ensure that only one SSH key pair and its corresponding object are generated for the entire cluster? Alternatively, do we need to copy multiple SSH public keys to the LPAR? Would this require changing the existing logic?

[littlejawa]

@bpradipt, @savitriH, sorry answering late here.
I've looked at the PR and played with it a little. Here are my thoughs:

• I feel the libvirt pool/vol creation should be part of the libvirt image handler, for consistency with other providers

• For the SSH key generation, I'm not sure how the key is used actually.
  For libvirt it is the authentication method that cloud-api-adaptor uses to access the hypervisor.
  But in general, is it just used in the podvm image as a way to ssh into it, assuming the ssh server was enabled there?
  Or are there other use cases?

  To log into the podvm we need to have the private key, so if it is generated in the cluster and available only there, it becomes less easy to troubleshoot the podvm.
  Unless we want to log to the CAA pod first (which has access to the key I guess)?
  Is that the intended troubleshooting path?

  If that's the case, I feel we should have this part of the code (generating keys+creating ssh-key-secret) grouped with other providers so that we can automate that task for everybody, not just libvirt.

• For configmap, the work done here simplifies the creation for other providers. Would it make sense to add libvirt support there?
  I understand this does not automate things as much as you're proposing here, but if we have this script available and working for all providers, maybe we can then automate it for everybody?