Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

clh: keep the stdout/stderr open for CLH until its process exits #2439

Closed
likebreath opened this issue Feb 4, 2020 · 2 comments · Fixed by #2648
Closed

clh: keep the stdout/stderr open for CLH until its process exits #2439

likebreath opened this issue Feb 4, 2020 · 2 comments · Fixed by #2648
Assignees
@likebreath @egernst @jcvenegas @sboeuf
Labels
enhancement Improvement to an existing feature medium-priority Urgent issue (resolve before unprioritised issues) needs-review Needs to be assessed by the team.

Comments

@likebreath
Copy link
Contributor

Which feature do you think can be improved?

The kata-runtime now captures cloud-hypervisor (CLH) output (stdout/stderr) to a local buffer cloudHypervisor::cmdOutput when launching CLH. The kata-runtime will exit when CLH launched successfully, and will leave CLH's output closed (not redirected) afterwards.

This issue can cause unexpected behaviors from CLH, e.g. causing CLH (child thread) panic while CLH tries to log warnings (to stderr) and stderr is not open (cloud-hypervisor/cloud-hypervisor#703).

How can it be improved?

One potential solution is, instead of redirecting to a local buffer that lives with kata-runtime, CLH's output can be redirected to a file, e.g. /dev/null for production, or a log file for debugging.

Additional Information

Please share your thoughts and advises.
@likebreath likebreath added enhancement Improvement to an existing feature needs-review Needs to be assessed by the team. labels Feb 4, 2020
@likebreath likebreath mentioned this issue Feb 4, 2020
Closed
@grahamwhaley
Copy link
Contributor

I'm prone to say you might want to capture the output, even in a production deployment. The end users might want to import those into their logging stack for post-analysis etc.
I don't know the wrinkles of how/where is going to be best to re-direct/capture/push that data in the kata runtime stack - but, I reckon @jodh-intel will have an idea (basically, whatever route we do today for qemu etc.).

@amshinde
Copy link
Member

amshinde commented Feb 6, 2020

Based on the debug settings for the hypervisor in our config.toml we can either have this redirected to a log file or /dev/null when debug is not enabled.

@jcvenegas jcvenegas added the medium-priority Urgent issue (resolve before unprioritised issues) label Mar 27, 2020
jcvenegas added a commit to jcvenegas/runtime that referenced this issue Apr 27, 2020
@jcvenegas
clh: Improve hypervisor logging
da4a76c 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/runtime that referenced this issue Apr 27, 2020
@jcvenegas
clh: Improve hypervisor logging
1bbfaad 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
@jcvenegas jcvenegas mentioned this issue Apr 27, 2020
Merged
jcvenegas added a commit to jcvenegas/runtime that referenced this issue Apr 27, 2020
@jcvenegas
clh: Improve hypervisor logging
04c3bf0 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/runtime that referenced this issue Apr 27, 2020
@jcvenegas
clh: Improve hypervisor logging
13f48b3 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/runtime that referenced this issue Apr 28, 2020
@jcvenegas
clh: Improve hypervisor logging
47bcdb9 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/runtime that referenced this issue Apr 29, 2020
@jcvenegas
clh: Improve hypervisor logging
c1d823a 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/runtime that referenced this issue Apr 30, 2020
@jcvenegas
clh: Improve hypervisor logging
c64565b 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/runtime that referenced this issue May 4, 2020
@jcvenegas
clh: Improve hypervisor logging
258e795 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/runtime that referenced this issue May 5, 2020
@jcvenegas
clh: Improve hypervisor logging
540c489 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/runtime that referenced this issue May 6, 2020
@jcvenegas
clh: Improve hypervisor logging
072463f 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/runtime that referenced this issue May 13, 2020
@jcvenegas
clh: Improve hypervisor logging
d31bd86 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/runtime that referenced this issue May 13, 2020
@jcvenegas
clh: Improve hypervisor logging
e1ee00d 
Use systemd-cat to collect hypervisor output. The `systemd-cat` program
will open a journal fd and call `cat(1)` to redirect all the output to
the fd. This requires an extra binary to read from hypervisor stdout
(that has combined stdin, stderr and serial terminal). But because it is
cat the overhead is minimal and only is started on Kata debug mode.

Depends-on: github.com/kata-containers/packaging#1047

Fixes: kata-containers#2439

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@likebreath likebreath

@egernst egernst

@jcvenegas jcvenegas

@sboeuf sboeuf

Labels
enhancement Improvement to an existing feature medium-priority Urgent issue (resolve before unprioritised issues) needs-review Needs to be assessed by the team.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

clh: Improve logging for cloud hypervisor jcvenegas/runtime
6 participants
@grahamwhaley @amshinde @likebreath @egernst @jcvenegas @sboeuf