Unable to start container using low memory limit

[chrischdi]

Issue Report

Bug

Container Linux Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1967.6.0
VERSION_ID=1967.6.0
BUILD_ID=2019-02-12-2138
PRETTY_NAME="Container Linux by CoreOS 1967.6.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr

Environment

We're running in Openstack / VMWare Integrated Openstack:

$ cat /proc/meminfo  | grep MemTotal
MemTotal:        8171244 kB

Expected Behavior

Docker container starts

Actual Behavior

Docker fails to start a container

Reproduction Steps

1. Start docker container including memory limit:

   docker run --rm --memory 10485760 -ti jimmidyson/configmap-reload:v0.2.2
   

/run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused "process_linux.go:367: setting cgroup config for procHooks process caused \"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/docker/aa95f28a0ecfc178c4fb52e361281f936f2e556679f55ae193aacd5b150a6416/memory.limit_in_bytes: device or resource busy\""": unknown.

When using a higher memory limit, starting the container is no problem.
When using a slightly higher memory limit, the result is varying:

$ docker run --rm --memory 12534336 -ti jimmidyson/configmap-reload:v0.2.2
/run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused "process_linux.go:367: setting cgroup config for procHooks process caused \"failed to write 12534336 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/docker/5ea6580804d337da46c06f4c06b4df7dce62adb17362687ac3d0faabe966f302/memory.limit_in_bytes: device or resource busy\""": unknown.
$ docker run --rm --memory 12534336 -ti jimmidyson/configmap-reload:v0.2.2
/run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: docker-runc did not terminate sucessfully: panic: No error following JSON procError payload. [recovered]
panic: No error following JSON procError payload.

goroutine 1 [running]:
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction.func1(0xc420145870)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:478 +0x23d
panic(0x55f49ba9ce40, 0x55f49bb1d5c0)
/usr/lib/go1.10/src/runtime/panic.go:502 +0x22d
github.com/opencontainers/runc/libcontainer.parseSync(0x55f49bb1f500, 0xc4200da108, 0xc420144a50, 0x3, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/sync.go:99 +0x2f3
github.com/opencontainers/runc/libcontainer.(*initProcess).start(0xc4200e0700, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/process_linux.go:323 +0x3b4
github.com/opencontainers/runc/libcontainer.(*linuxContainer).start(0xc4201ae000, 0xc4200e6d80, 0x1, 0x0, 0xc420144c50)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/container_linux.go:343 +0x95
github.com/opencontainers/runc/libcontainer.(*linuxContainer).Start(0xc4201ae000, 0xc4200e6d80, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/container_linux.go:237 +0xc5
main.(*runner).run(0xc420145398, 0xc4200e49c0, 0x0, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/utils_linux.go:303 +0x90c
main.startContainer(0xc420124140, 0xc4201260e0, 0x1, 0x0, 0x0, 0x20, 0xc4200d2cf0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/utils_linux.go:428 +0x3c4
main.glob..func2(0xc420124140, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/create.go:65 +0xb0
reflect.Value.call(0x55f49baa85c0, 0x55f49bb1cb10, 0x13, 0x55f49b7d9e57, 0x4, 0xc420055830, 0x1, 0x1, 0xc4200600c0, 0x55f49bb10e20, ...)
/usr/lib/go1.10/src/reflect/value.go:447 +0x96b
reflect.Value.Call(0x55f49baa85c0, 0x55f49bb1cb10, 0x13, 0xc420055830, 0x1, 0x1, 0x55f49b79d4b8, 0x55f49b7d9f77, 0x4)
/usr/lib/go1.10/src/reflect/value.go:308 +0xa6
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction(0x55f49baa85c0, 0x55f49bb1cb10, 0xc420124140, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:487 +0x16f
github.com/opencontainers/runc/vendor/github.com/urfave/cli.Command.Run(0x55f49b7da66b, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55f49b7de64e, 0x12, 0x0, ...)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go:191 +0x9ac
github.com/opencontainers/runc/vendor/github.com/urfave/cli.(*App).Run(0xc420102180, 0xc4200100f0, 0xf, 0xf, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:240 +0x548
main.main()
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/main.go:147 +0xaea
: unknown.
$ docker run --rm --memory 12534336 -ti jimmidyson/configmap-reload:v0.2.2
2019/02/18 07:16:16 Missing volume-dir
2019/02/18 07:16:16
Usage of /configmap-reload:
-volume-dir value
the config map volume directory to watch for updates; may be used multiple times
-webhook-method string
the HTTP method url to use to send the webhook (default "POST")
-webhook-status-code int
the HTTP status code indicating successful triggering of reload (default 200)
-webhook-url string
the url to send a request to when the specified config map volume directory has been updated

### Other Information ###

Unable to reproduce the problem on same os configuration but using the following CoreOS Release:

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1967.3.0
VERSION_ID=1967.3.0
BUILD_ID=2019-01-08-0044
PRETTY_NAME="Container Linux by CoreOS 1967.3.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

We did not test `1967.5.0`

[chrischdi]

Also tested 1967.5.0: so it looks to me that the runc fix is producing this bug.
Because I can also reproduce the problem on 1967.5.0

One more note: we are using docker 18.06.1-ce, installed via torcx.

[dongsupark]

You would probably want to look into the upstream issue, which is still in discussions: opencontainers/runc#1980

[chrischdi]

Hi again,

the problem should be resolved by opencontainers/runc#1984 and opencontainers/runc#2006 .
Will these patches also get into a new CoreOS Release?