-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"runc init" goes into a loop on RHEL 7.6 (runc 09c8266bf2fcf9519a651b04ae54c967b9ab86ec) #1988
Comments
Original issue containerd/containerd#3027 |
An strace of the looping "runc init": strace log
|
To me this looks like a RHEL kernel bug -- |
@cyphar do you have a code fragment that I can run to test that ? Alternatively, a flag that disables this functionality would allow runc to continue until the causing issue is fixed. |
#1984 adds a "flag" (rather an environment variable) to disable A simple code fragment would be something like the following (it should not give you any failures if Check script.#define _GNU_SOURCE
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/syscall.h>
/* Use our own wrapper for memfd_create. */
#if !defined(SYS_memfd_create) && defined(__NR_memfd_create)
# define SYS_memfd_create __NR_memfd_create
#endif
/* memfd_create(2) flags -- copied from <linux/memfd.h>. */
#ifndef MFD_CLOEXEC
# define MFD_CLOEXEC 0x0001U
# define MFD_ALLOW_SEALING 0x0002U
#endif
int memfd_create(const char *name, unsigned int flags)
{
#ifdef SYS_memfd_create
return syscall(SYS_memfd_create, name, flags);
#else
errno = ENOSYS;
return -1;
#endif
}
/* This comes directly from <linux/fcntl.h>. */
#ifndef F_LINUX_SPECIFIC_BASE
# define F_LINUX_SPECIFIC_BASE 1024
#endif
#ifndef F_ADD_SEALS
# define F_ADD_SEALS (F_LINUX_SPECIFIC_BASE + 9)
# define F_GET_SEALS (F_LINUX_SPECIFIC_BASE + 10)
#endif
#ifndef F_SEAL_SEAL
# define F_SEAL_SEAL 0x0001 /* prevent further seals from being set */
# define F_SEAL_SHRINK 0x0002 /* prevent file from shrinking */
# define F_SEAL_GROW 0x0004 /* prevent file from growing */
# define F_SEAL_WRITE 0x0008 /* prevent writes */
#endif
#define RUNC_MEMFD_COMMENT "runc_cloned:/proc/self/exe"
#define RUNC_MEMFD_SEALS \
(F_SEAL_SEAL | F_SEAL_SHRINK | F_SEAL_GROW | F_SEAL_WRITE)
#define bail(msg) \
do { perror(msg); exit(1); } while (0)
int main(void)
{
int fd = memfd_create(RUNC_MEMFD_COMMENT, MFD_CLOEXEC | MFD_ALLOW_SEALING);
if (fd < 0)
bail("memfd_create failure");
if (fcntl(fd, F_ADD_SEALS, RUNC_MEMFD_SEALS) < 0)
bail("f_add_seals failure");
int seals = fcntl(fd, F_GET_SEALS);
if (seals < 0)
bail("f_get_seals failure");
if (seals != RUNC_MEMFD_SEALS)
bail("f_get_seals incorrect result");
return 0;
} |
Compiled with "cc -o checkseal checkseal.c", RC=0 (both as root and as a normal(*) user)
|
The reason why it's infinite looping is because it keeps copying the binary, because it thinks that
There really isn't another explanation. It's possible that there is a permission issue with SELinux that makes it not want to give |
Upgraded to containerd.io 1.2.2-3.3
An lsof of the "runc init" shows (Note /memfd:runc_cloned:/proc/self/exe (deleted) on 3rd entry):
|
This looks reasonable, and matches what I was thinking:
Have you tried to run with SELinux disabled, or checked whether there was an AVC denial logged? Then again, the code definitely works on Fedora (I tested the mitigation on Fedora with SELinux before the patch was posted). |
Looking at my patch in #1984 it actually looks like I already have a solution for this issue without needing to deal with the bad RHEL backport (as part of the |
@cyphar I'm uncertain on how to use git to pull the correct version into my machine. Can you help me ? |
I'm not sure if you already have a working Go environment set up, but if you don't here is how you'd do it:
And then you fetch
Then fetch my PR:
And then you can build it:
And finally, install it -- you want to replace the |
Went well until the make:
|
Installed packages with "go get -v github.com/urfave/cli github.com/containerd/console github.com/coreos/go-systemd/activation github.com/opencontainers/runc/libcontainer github.com/opencontainers/runc/libcontainer/cgroups github.com/opencontainers/runc/libcontainer/cgroups/systemd github.com/opencontainers/runc/libcontainer/configs github.com/opencontainers/runc/libcontainer/configs github.com/opencontainers/runc/libcontainer/intelrdt github.com/opencontainers/runc/libcontainer/nsenter github.com/opencontainers/runc/libcontainer/specconv github.com/opencontainers/runc/libcontainer/specconv github.com/opencontainers/runc/libcontainer/system github.com/opencontainers/runc/libcontainer/user github.com/opencontainers/runc/libcontainer/utils github.com/opencontainers/runtime-spec/specs-go github.com/docker/go-units" Now make yields:
|
I made a mistake in my instructions, you should clone This should fix your issue:
|
Now it compiles. Replaced /usr/sbin/runc with the new image. runc seems to run ok now (0.09s user+sys to exec an alpine container). |
Did some more tests and looks like it's enjoying itself. :-) What does the path forward look like ? Especially getting this translated into docker-ce, docker/for-linux, docker/runc and containerd & friends ? (I'm still trying to figure out the workflow) |
Since runc is a separate binary (and completely separate project) to containerd and Docker, they have very little to do. Once it gets merged, then Docker will presumably update the pinned version they're using for their releases and so it'll be available in the next version (of |
The https://github.com/docker/docker-ce repository doesn't have issues. Note: I'm still learning to use git/github, and the intersection of git & multiple projects is still a bit overwhelming, so: |
Wait until #1984 is merged, and then you can send an update to https://github.com/moby/moby (which is where Docker is actually developed).
You can leave it open, it'll be automatically closed when #1984 is merged. |
Correct; yes,
This issue (as well as some other issues related to the runc CVE fix) is being tracked internally at Docker, and will be included in a future update. Depending on timing, that may either be as part of a new release of containerd, or as a packaging-only change for the containerd.io package. |
Independent of the fix here in runc; even though the patch that's being worked on here would address the problem, there still looks to be a bug in the RHEL kernel. I think that bug should be reported with Red Hat so that it doesn't get lost; @pinacoelho if you have an active Red Hat subscription, could you open a ticket with them? https://bugzilla.redhat.com/index.cgi /cc @vbatts perhaps you know if this problem is already tracked? |
I cannot replicate this with kernel Tracing shows
ie working as expected. |
@justincormack Are you testing on the RHEL version of those kernels or the CentOS ones? My experience has shown that RHEL backports can be very different to CentOS ones. |
@kolyshkin Managed to reproduce it in #1984. |
I'm trying to figure out a minimal testcase to send to redhat. I expect it to run ok in cosmic cuttlefish (kernel V4) and rhel 7.6. Trying not to get lost: |
Sure, but the problem appears to be the |
@pinacoelho if that helps, I filed a bug to red hat, using the kernel selftest code: https://bugzilla.redhat.com/show_bug.cgi?id=1679829 |
I’m having problems recovering my RH password. Also failing to compile the
testcase.
Thank you for opening the bug report, and I’ll feedback as soon as I
resolve either issue.
…On Thu, 21 Feb 2019 at 23:39, Kirill Kolyshkin ***@***.***> wrote:
I'm trying to figure out a minimal testcase to send to redhat.
@pinacoelho <https://github.com/pinacoelho> if that helps, I filed a bug
to red hat, using the kernel selftest code:
https://bugzilla.redhat.com/show_bug.cgi?id=1679829
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1988 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/APtnSD2PIOB_2anvFqsW2sKSDd-DA0tWks5vPy4qgaJpZM4bBSXg>
.
|
To be clear, the runc build that is having an issue, is it the runc package provided with RHEL? |
No, it’s the runc from the docker-ce repository.
…On Fri, 22 Feb 2019 at 13:44, Vincent Batts ***@***.***> wrote:
To be clear, the runc build that is having an issue, is it the runc
package provided with RHEL?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1988 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/APtnSLG9GL8b3goPeHtah6v29wv19ZfOks5vP_QzgaJpZM4bBSXg>
.
|
Have we confirmed that runc master has the issue? I've double checked this with the dev and test teams, and they're not seeing this loop. The kernel passing a test-suite is another issue, and I'm glad that has been filed separately. |
|
Confirming that the following environment has the issue as well as I have only seen people saying "RHEL" in the various issues about this. CentOS Linux release 7.6.1810 (Core) docker run hello-world just goes nuts looping as described. |
I've tested this patch on centos(7.3 to 7.6) and rhel(7.3 and 7.6), all goes well without going looping. You can reproduce this to get a vagrant rhel7 box (e.g. from https://app.vagrantup.com/generic/boxes/rhel7), run a vm and install docker and all other packages from centos7 repo.
I noticed that your
Although I use centos yum repo as rhel repo mirror, the kernel is from rhel. AFAIK, centos build their kernel form rhel kernel source(you can get it from rhel kernel srpm) with only a little change. May well not the kernel side cause this issue. |
@kolyshkin @cyphar @vbatts https://gist.github.com/pinacoelho/396ef3302ae60b17662e29fd57172953 |
@pinacoelho cool. I linked it in the bugzilla for the mem_fd on rhel. |
Just a FYI: The runc included in containerd.io 1.2.5-3.1.el7 is working correctly with that kernel. runc version 1.0.0-rc6+dev https://gist.github.com/pinacoelho/396ef3302ae60b17662e29fd57172953 still shows seals getting lost after the pexecve. |
mark |
After the last upgrade, docker hangs while starting containers.
htop shows the runc processes consuming 100% of the CPU time.
Environment
RHEL 7.6 (up to date)
from https://download.docker.com/linux/centos/7/x86_64/stable:
containerd.io-1.2.2-3.3.el7.x86_64.rpm 2019-02-11 16:03:53 22.1 MiB
docker-ce-cli-18.09.2-3.el7.x86_64
docker-ce-18.09.2-3.el7.x86_64
Steps to reproduce the issue:
Describe the results you received:
"docker run alpine" hangs. htop shows the "runc init" process consuming 100% cpu.
Describe the results you expected:
Expected docker to run the alpine container and return.
Output of
containerd --version
:Note that containerd --version outputs the same result for 1.2.2-3 and 1.2.2-3.3
Downgrading containerd.io to containerd.io-1.2.2-3.el7.x86_64.rpm 2019-01-09 21:07:30 solves the issue.
runc --version from 1.2.2-3 (GOOD):
runc --version from 1.2.2-3.3 (BAD):
** Delta between versions **
96ec217...09c8266
Kernel level: Linux moykano 3.10.0-957.5.1.el7.x86_64 Change version to 0.x #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux (rpm: kernel-3.10.0-957.1.3.el7.x86_64)
lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: RedHatEnterpriseWorkstation
Description: Red Hat Enterprise Linux Workstation release 7.6 (Maipo)
Release: 7.6
Codename: Maipo
The text was updated successfully, but these errors were encountered: