libwebsockets fails inside a container: OOM allocating 1073741816 fds

[drigz]

I observed this when running mosquitto inside a container, using containerd 1.5.5:

/ # /usr/sbin/mosquitto -v -c /mosquitto/config/mosquitto.conf 
1634294021: mosquitto version 1.6.12 starting                                                            
1634294021: Config loaded from /mosquitto/config/mosquitto.conf.
1634294021: Opening websockets listen socket on port 9090.
1634294021: libuv support not compiled in         
1634294021: OOM allocating 1073741816 fds           
1634294021: ZERO RANDOM FD                      
1634294021: Error: Unable to create websockets listener on port 9090.

The issue appears to be caused by containerd/containerd#4475, which sets LimitNOFILE=infinity, apparently in favor of container-local accounting. However, kubernetes/kubernetes#3595 prevents me from easily avoiding this crash. It can be worked around by running ulimit -n 1024 before starting mosquitto.

This was reported before in #1769, but closed. I've reopened because I expect more people will run into this as newer versions of containerd are more widely used.

[lws-team]

Lws works fine "inside a container", many of the CI platforms are systemd-nspawns. Your container has a munged ulimit -n is the issue.

Lws lets you set the process ulimit -n (aka, "nofiles") by setting info.rlimit_nofile.

           int rlimit_nofile;
        /**< 0 = inherit the initial ulimit for files / sockets from the startup
         * environment.  Nonzero = try to set the limit for this process.
         */

and since cab6309 in Dec 2020 lws has a workaround for unreasonable ulimit reported by the OS, it restricts it to 2560.

So how old is your lws?

[drigz]

Thanks for the pointer and apologies for the imprecise issue title. mosquitto 1.6.12 pins lws 2.4.2 (source). This was bumped to 4.2.0 in May, and I can't reproduce the issue with mosquitto 1.6.15.

I would argue that the high limit is not unreasonable; https://man7.org/linux/man-pages/man2/getrlimit.2.html doesn't guarantee that a process can allocate that many FDs or that much memory, just "This specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE."

Either way, the case of LimitNOFILE=infinity is fixed, so I'll close this. Sorry for the noise!