Troubleshooting an Out Of Memory (OOM) failure when attempting to launch a Pelican Origin

[aowen-uwmad]

Was testing how to set up and run a Pelican Origin service using the recommended Docker containers and encountered a seemingly intractable error.
With Logging.Level = "debug":

$ docker run --rm -it     \
    --name origin \
    -p 8444:8444 -p 8443:8443 \
    -v $(pwd)/config/pelican.yaml:/etc/pelican/pelican.yaml \
    -v $(pwd)/origin-data:/data \
    hub.opensciencegrid.org/pelican_platform/origin:v7.8.7 \
    serve -p 8444

<...>

INFO[2024-06-17T16:06:15Z] Config maximum number of connections restricted to 178956970  daemon=xrootd.origin
ERROR[2024-06-17T16:06:19Z] xrootd.origin process failed unexpectedly: signal: killed
Error: context canceled
INFO[2024-06-17T16:06:19Z] Gracefully stopping origin TTL cache eviction...
ERROR[2024-06-17T16:06:19Z] Fatal error occurred at the start of the program. Cleanup started: context canceled
INFO[2024-06-17T16:06:19Z] Director health test timeout loop has been terminated
INFO[2024-06-17T16:06:19Z] Xrootd metrics cache eviction has been stopped
INFO[2024-06-17T16:06:19Z] Origin TTL cache eviction has been stopped
ERROR[2024-06-17T16:06:19Z] Fatal error occurred that lead to the shutdown of the process: xrootd.origin process failed unexpectedly: signal: killed

After many attempts at changing the configuration and how the container was launched, was eventually able to determine this error was being caused by xrootd being OOM killed during startup. (Thanks @jhiemstrawisc , @haoming29 for the assist.)

Eventually we were able to reproduce the error on a different machine by manually controlling the amount of memory the Docker container was allowed to use via the -m option of docker run. With some empirical testing, determined that this particular container/execution needed > ~5GB of RAM in order to function without throwing the errors shown above.
Reprovisioning the VM that was initially being used with more memory resolved the error.

From our discussion, sounded like their were two things to consider moving forward:

1. Investigating why xrootd needs that memory, since the collective intuition is that it doesn't.
2. Have Pelican detect when xrootd has been OOM killed and report a better error message than the above messages.