Discuss introducing a debug console facility

[jodh-intel]

Background

We have considered adding a debug console in Clear Containers in the past:

• WIP: Debug: Enable VM debug console clearcontainers/runtime#658
• Debug: Provide a way to enable a debug console to the VM clearcontainers/agent#122
• Adding a minimal shell to default rootfs clearcontainers/osbuilder#36

However, discussions stalled for two main reasons:

Security

The concern that introducing such a feature could be an attack vector.

Image size

Adding a shell (even a tiny one) is going to bloat the images slightly and that space isn't going to be used 99.99% of the time.

Rationale

It would be very useful for developers and admins to have the ability to debug a running container from the guest-side root namespace. Note that docker exec is NOT what we want as that is not running in the guest root namespace and is thus constrained.

Image support

We could generate two images - one with a shell and one without. However, the general view is that this is suboptimal since:

• The "debug-enabled" version would probably get less testing.
• If a developer/admin needs to debug a container that is already running, they might not be running the "debug" image which defeats the point of the feature.

Architecture

• runv expects the agent to be running as PID = 1 (init daemon)

  A debug console shell would either need to run as a separate thread or a child process of the agent.

• cc-runtime (virtcontainers) assumes the agent to be running with PID != 1

  A debug console shell can be launched by the init daemon (systemd by default) or could be handled as a separate thread / child of the agent.

From a testing (and security) perspective, it would be safer to have a single code path for a debug console.

Configuration and logging

If we introduce a debug console facility it:

• MUST be disabled by default.
• MUST be possible to query the status of the feature somehow to be assured that it is disabled.
• MUST generate an agent log entry when the feature is enabled to allow an admin to detect this change.
• MUST generate an agent log entry when the feature is used to allow an admin to track usage.

[jodh-intel]

/cc @mcastelino, @sboeuf, @jcvenegas.

[jodh-intel]

/cc @egernst.

[jodh-intel]

Any thoughts on this @gnawux, @bergwolf?

[gnawux]

runV/hyperstart had a method to run a process outside any container. I agree this is useful sometimes.

add @laijs

[sboeuf]

@jodh-intel I think we all agree this is really needed since we want to be able to access a VM to perform some advanced debugging.

I am not sure if we need to maintain an official debug image, instead we could provide the right documentation on how a user can create its own debug image based on the official one with osbuilder.
And also we need the documentation to explain how we should disable the debug logs from the proxy, so that we can actually connect to console.sock from a socat ... command.

Unfortunately, and as you mentioned for security purposes, we should not have a dedicated way to debug from the official image itself.

[grahamwhaley]

If possible, we could add a flag to osbuilder to automatically add and configure the bits to an image - that would make the process of generating a debug image a little easier as well.

[jodh-intel]

@mcastelino can comment further, but to be really useful, an admin would be able to debug any pod. That implies a single image type which either contains some sort of built-in but dormant debug console facility that can be enabled as desired [*], or a way to allow a debug console to be added securely by an admin to the running virtual machine.

Maybe a compromise would be to provide some way to do the following (optionally after some sort of clone operation so the debug fiddling could be performed on a copy of the real workload?):

• pause the pod.
• save the state.
• switch to a new image (same release version, but with debug machinery enabled).
• restore the state.
• enable debug console.
• start debug console.
• (optionally) unpause the pod.

---

[*] - but must be disabled by default when new containers are created.

[sboeuf]

@jodh-intel not sure I understand why we need such a complex functioning ? The user will just need to make sure the proxy(or anything else) is not consuming the dedicated socket, but other than that, it can be joined anytime, no need to pause the VM for that.

[jodh-intel]

Hi @sboeuf - Oh, sorry - I hadn't read you previous comment correctly.

I think the confusion here revolves around perceived requirements. I think we all agree that:

• we want such a facility.
• we require it to be disabled by default.

But what still doesn't seem very clear is whether we have the following requirement:

• we will create a single official image that provides a debug facility which is disabled by default.

Folk with a security hat on will tend to say "no", whilst folks who want to be able to debug running container issues will tend to say "yes".

Back to your corners y'all for a wet sponge and towel... and...

🔔 Round 1! 🔔

[grahamwhaley]

my 2p - maybe you can have a single image if you can have access to the debug port secure enough that it does not pose a major problem during deployment.
One suggestion maybe is to have the port/socket/terminal/whatever secured with a key that is only accessible to root and/or the container infra. For instance, if you expose a socket tty, make it SSH and store the key in the container manager or orchestration side. If that is compromised already then your container is already compromised etc.

[sboeuf]

Having the terminal + ssh into the image will make it bigger, should not be that big though, but might worth to take into account.
We have all our images being created from osbuilder and I think, as mentioned by @grahamwhaley, having a simple flag to generate a debug image (including what's needed) should be easy. 99% of the time, an issue discovered on a specific packaged image should be easily reproduced with an image generated from osbuilder.

Now, I agree that having the debug capability included in every image would be easier because we would never have the problem of not being able to reproduce the issue through a specific debug build.

Maybe the alternative to this would be to look at the problem upside down here, and consider having an image including debug by default, but only when people would need to use Kata in production, they would generate a "production-ready" image from osbuilder. This image would be the current image that we have (no debug included).

[mcastelino]

I am not sure if we need to maintain an official debug image, instead we could provide the right documentation on how a user can create its own debug image based on the official one with osbuilder.
And also we need the documentation to explain how we should disable the debug logs from the proxy, so that we can actually connect to console.sock from a socat ... command.

@sboeuf I do not agree that the user should generate the image. It is not hard for us to generate the image each time we create a release and then the user can simply switch to the debug image by modifying the toml. @egernst and myself have had to support multiple customers and developers to get this working.

Making our users and customer jump through hoops just to be able to debug is not a productive use of their time or ours.

[sboeuf]

@mcastelino I don't have a strong opinion on that, and based on your experience with customers, I trust your opinion about this !

[jodh-intel]

@mcastelino - what's your view though on whether we should provide a single image or two ("standard" and "debug")?

[mcastelino]

@jodh-intel two images generated by our build system would be nice. And we could only choose to only validate the standard image.

And the debug image is just there for interactive debug convenience. As we cannot flip the interactive debuggability on, on the fly without opening up potential security issues; having a separate debug image may be better from a security and validation point of view. The only test we need to run on the debug image is to verify if the console is accessible.

Also the debug image does not really helpful in the cases where the POD panics or the VM crashes. That has to be debugged via kernel, proxy and shim logs. We have found the debug image useful typically when we deploy a new/customized QEMU or Kernel and things do not quite work as expected.

@sboeuf we just need to include a minimal shell. There is no need for ssh, as the console will not need auth. That is unless we change the way we provide debug or if we want to provide a single image.

[sboeuf]

@mcastelino I am fine with this solution :)
FYI, I have been using osbuilder, but instead of building a clearlinux rootfs, I have been using a fedora one, and I have realized that /usr/bin/bash is already there. Pretty useful for debugging !

@jodh-intel are we planning to test only images based on clearlinux rootfs or should we also validate other images based on different distros ?

[jodh-intel]

As we cannot flip the interactive debuggability on, on the fly without opening up potential security issues

@mcastelino - you mean since theoretically if you want to debug container X, the debug facility would also be available to container Y in the same pod?

Also, I'm slightly confused. I had thought that you and @egernst were in favour of having the ability to debug any container. Have I misunderstood or has that requirement now changed?

Also the debug image does not really helpful in the cases where the POD panics or the VM crashes. That has to be debugged via kernel, proxy and shim logs.

Right. All debug logs are now tested on all PRs using the kata-log-parser tool. Plus all system components now crash "properly":

• core dump if debug is enabled.
• always log their stacktraces to the system journal (unless they were SIGKILL'd).

I've added tests for these things on the issue I raised for creating a "packaging CI": kata-containers/ci#10.

@sboeuf - that's a good question. Given the amount of boot optimisation Clear Linux has performed, we want to ensure it remains a first class citizen here imho. However, if we have CI capacity we should consider testing a few variants (maybe centos and ubuntu [once we have that support]?)

wdyt @xu, @bergwolf, @chavafg, @jcvenegas, @grahamwhaley?

[grahamwhaley]

I think it primarily comes down to:

• test capacity - do we have the resource and are we willing to take the time hit on PR CI feedback
• how much would we test each rootfs variant - the full test suite?
• which variants to test - one half of me says things like Alpine (small cloud distros), the other half says RH, Ubuntu et. al. (larger distros that we are fairly sure may be used as the rootfs in the relevant parent distros).

[jodh-intel]

Yep. I think we should aim to test >1 images just to minimise us inadvertently making assumptions about the image, but given the flexibility of the system, we can't realistically test all images.

However, we can enhance our existing osbuilder tests to create a basic container using each supported image type. I've raised kata-containers/osbuilder#68 to do that.

[egernst]

Refreshing this -- I think one thing we can agree on, based on reading last few comments, is that it makes sense to provide a second image which is identical to the primary, but has this debug console enabled (ie, has the shell, has the service). Is this something that we can move forward with? I assume this would be included in the packages provided. @erick0z