workerd crashed in load test

[Yamabuki-bakery]

I am using ab and wrk command to do load test on workerd and it crashed, I copied the config file of the hello world sample from this repo.

The output shows:

$ ./workerd serve config.capnp --verbose --watch
workerd/util/symbolizer.c++:98: warning: Not symbolizing stack traces because $LLVM_SYMBOLIZER is not set. To symbolize stack traces, set $LLVM_SYMBOLIZER to the location of the llvm-symbolizer binary. When running tests under bazel, use `--test_env=LLVM_SYMBOLIZER=<path>`.
*** Fatal uncaught kj::Exception: kj/async-io-unix.c++:1408: overloaded: accept: Too many open files
stack: ./workerd@18b681b ./workerd@18b9670 ./workerd@18b9df9 ./workerd@18ba514
root@yuina:~/workerd# ./workerd serve config.capnp --verbose --watch
workerd/util/symbolizer.c++:98: warning: Not symbolizing stack traces because $LLVM_SYMBOLIZER is not set. To symbolize stack traces, set $LLVM_SYMBOLIZER to the location of the llvm-symbolizer binary. When running tests under bazel, use `--test_env=LLVM_SYMBOLIZER=<path>`.
*** Fatal uncaught kj::Exception: kj/async-io-unix.c++:1408: overloaded: accept: Too many open files
stack: ./workerd@39eb01d ./workerd@39ea7b2 ./workerd@18b681b ./workerd@18b9670 ./workerd@18b9df9 ./workerd@18ba514 ./workerd@1862ecd ./workerd@18672a7 ./workerd@1867024 ./workerd@186700c ./workerd@3a17d2e ./workerd@3a1792d ./workerd@3a15c68 ./workerd@3a15a2a ./workerd@1858299 /lib/x86_64-linux-gnu/libc.so.6@29d8f /lib/x86_64-linux-gnu/libc.so.6@29e3f ./workerd@185802d

The wrk command:

$ wrk -c 1500 -d 10s -t 2 http://192.168.1.100:8080

The OS is Ubuntu 22.04, uname -a shows:

Linux yuina 5.15.0-87-generic #97-Ubuntu SMP Mon Oct 2 21:09:21 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

It seems that the exception on sockets is not handled correctly, does anyone has ideas?

[kentonv]

This is because workerd hit the limit on total open file descriptors (which includes network connections). This is a common system configuration issue that applies to all servers -- it's not specific to workerd. If you're running workerd in a shell, you can usually use the shell's ulimit command to increase the limit, although you may need to be root to increase the limit beyond a certain point. If you're running workerd under some sort of server runner like systemd, you'll need to configure it to provide a higher ulimit.

[Yamabuki-bakery]

Thank you very much! I also want to know if I can use Nginx to reverse proxy workerd, through unix socket to bypass this limit?

[kentonv]

That won't help, because:

1. nginx is subject to the same limits.
2. unix sockets also count against the limit.

If you have root permissions on the machine you can increase the limit to whatever you want. There's no need for elaborate work-arounsd here.