Better handle hitting the memory limit

[kolyshkin]

This improves runc error reporting in cases where memory limit is set too low.

NOTE that while this is clearly an enhancement I want this in rc94 because

• it does not change runc behavior other than error text;
• it helps a lot with some issues reported by customers.

1. Improve EBUSY handling on setting cgroup memory limit

On cgroup v1 + fs driver, setting memory.limit_in_bytes results in EBUSY. This is reported as

ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: failed to write "1000": write /sys/fs/cgroup/memory/user.slice/user-1000.slice/session-1011.scope/xe3/memory.limit_in_bytes: device or resource busy

To decipher the error a user needs to know that EBUSY is returned by the kernel when the limit being set is too low. In addition, it would be handy to know what the current usage is.

Handle EBUSY and report this:

ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: unable to set memory limit to 1000 (current usage: 1204224, peak usage: 2936832)

Related to #2736

2. Check for OOM kill on container start

When a container fails to start due to memory limit set too low, and container init being OOM-killed,
error messages returned by runc are semi-random and rather cryptic. Here are a few examples
(truncated to remove the common prefix for clarity):

• process_linux.go:348: copying bootstrap data to pipe caused: write init-p: broken pipe (cgroup v1 + systemd driver)
• process_linux.go:352: getting the final child's pid from pipe caused: EOF (cgroup v1 + systemd driver)
• process_linux.go:495: container init caused: read init-p: connection reset by peer (cgroup v2)
• process_linux.go:484: writing syncT 'resume' caused: write init-p: broken pipe (cgroup v2)

On container start error path, add a check if OOM kill has happened, and report it instead of the original (cryptic) error:

ERRO[0000] container_linux.go:367: starting container process caused: container init was OOM-killed (memory limit too low?)

(or, if --debug is set, also provide the original error):

ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:343: container init was OOM-killed (memory limit too low?) caused: process_linux.go:520: container init caused: process_linux.go:509: writing syncT 'resume' caused: write init-p: broken pipe

3. Check for OOM kill on container exec

Same as above, with some nuances:

1. The container is already running and OOM kill counter might not be
   zero. This is why we have to read the counter before exec and after
   it failed.

2. An unrelated OOM kill event might occur in parallel with our exec
   (and I see no way to find out which process was killed, except to
   parse kernel logs which seems excessive and not very reliable).
   This is why we report possible OOM kill.

The error message changed from

ERRO[0000] exec failed: container_linux.go:367: starting container process caused: read init-p: connection reset by peer

to

ERRO[0000] exec failed: container_linux.go:367: starting container process caused: process_linux.go:105: possibly OOM-killed caused: read init-p: connection reset by peer

4. Some minor improvements and refactoring along the way.

Please see individual commits for details.

TODO

• look into why runc init memory current and max differs that much
• look into why systemd v1 driver does not fail to set the limits like fs driver does see Fix discrepancy between systemd and fs cgroup managers #2813 / libct/cgroups/systemd: don't set limits in Apply #2814.

[kolyshkin]

Before:

ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: failed to write "1000": write /sys/fs/cgroup/memory/user.slice/user-1000.slice/session-1011.scope/xe3/memory.limit_in_bytes: device or resource busy

After:

ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: unable to set memory limit to 1000 (current usage: 1204224, peak usage: 2936832)

This at least gives some info about what's wrong as well as a hint about the lowest possible limit.

This also sheds some light about what is going on in #2736 -- apparently runc init has some peak memory use upon the start, this is why retrying on EBUSY helps @wzshiming.

Yet I am not convinced this should be handled by a retry.

[kolyshkin]

Failure on centos 7, seems like a random one (some rare race in old systemd?):

not ok 68 ps -e -x
(in test file tests/integration/ps.bats, line 53)
  `[ "$status" -eq 0 ]' failed
 runc spec (status=0):
 runc run -d --console-socket /tmp/console.sock test_busybox (status=1):
 time="2021-02-19T11:44:35Z" level=error msg="container_linux.go:367: starting container process
    caused: process_linux.go:502: container init
    caused: process_linux.go:465: setting cgroup config for procHooks process
    caused: Unit runc-test_busybox.scope is not loaded."

[kolyshkin]

@wzshiming can you test this PR on your setup to reproduce #2736 and copy-paste the runc errors here? It will help to understand what is going on in your case.

[kolyshkin]

No longer a wip. runc init memory usage analysis were not done, but it won't change anything in this code.

[kolyshkin]

Moved to draft to add OOM handling in runc exec. Added, tested, ready for the prime time!