Request to address zombie/defunct processes by modifying container main process

[andy108369]

Issue with sleep infinity in awesome-akash repo

Problem Description

In the awesome-akash repository, several containers are configured to run sleep infinity as their main process. This configuration can lead to unintended behavior, particularly the accumulation of zombie or defunct processes. This occurs because sleep infinity does not handle child processes properly, causing them to remain in a defunct state.

Here's an example where sleep infinity is used:

awesome-akash$ git grep -i sleep |grep infin
Ethereum_2.0/main.sh:sleep infinity
Falcon-7B/Dockerfile:CMD python3 falcon7b.py && sleep infinity
Sentinel-dVPN-node/main.sh:        sleep infinity
bitcoin/main.sh:sleep infinity
cryptodredge-c11/entrypoint.sh:sleep infinity
semantra/deploy.yaml:        sleep infinity ;'
softether-vpn/launch:sleep infinity

Impact

It is possible for certain deployments to initiate subprocesses that do not properly implement the wait() function. This improper handling can result in the formation of <defunct> processes, also known as “zombie” processes. Zombie processes occur when a subprocess completes its task but still remains in the system’s process table due to the parent process not reading its exit status. Over time, if not managed correctly, these zombie processes have the potential to accumulate and occupy all available process slots in the system, leading to resource exhaustion.

These zombie processes aren’t too harmful much (they don’t occupy cpu/mem / nor impact cgroup cpu/mem limits) unless they take up the whole process table space so no new processes will be able to spawn, i.e. the limit:

$ cat /proc/sys/kernel/pid_max
4194304

If sleep infinity is set as the main container process (PID 1), it fails to properly reap child processes, leading to their accumulation as zombie processes. Containers with such configurations may be terminated by the zombie killer cron job, implemented by some providers to handle these defunct processes.

Proposed Solutions

1. Use a Proper Primary Process: Containers should use their main application or a robust init system as the primary process. For example, running /usr/sbin/sshd -D is preferable to sleep infinity.
2. Process Management in Containers: Consider using dedicated init systems like tini, dumb-init, or runit, which are designed to handle child processes correctly:
   • tini
   • dumb-init
   • runit Dockerfile example
   • supervisord example by Docker.com
   • s6 example

Additional Resources

• Multi-service containers and managing child processes
• Phusion's baseimage as a solid init system
• Docker and the PID 1 zombie reaping problem
• Container Init Process

Request for Action

I suggest reviewing the current use of sleep infinity across the repository and discussing potential alternatives for better process management. This change could improve the stability and performance of deployments using this repository.

[andy108369]

I also thought that these days, most applications running on Kubernetes shouldn't need it. The pause container and sharing the process namespaces should be able to cover many responsibilities that tini/dumb-init had: https://www.ianlewis.org/en/almighty-pause-container

But having seen bunch of those zombie processes, this doesn't seem to be the case.
I couldn't find a quick reproducer that I could use a good example though, maybe need to bounce the pod to trigger it or so.

Dockerfile update would be:

• tini example

# Add Tini
ADD https://github.com/krallin/tini/releases/download/v0.19.0/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]

# Run your program under Tini
CMD ["/your/program", "-and", "-its", "arguments"]

---

FWIW: here is a Docker example of zombie processes:
https://github.com/jordanvrtanoski/zombie-docker-demo

To make Docker use tini, just add the --init flag:

docker run -ti --init --rm zombiedemo /zombie/ignore

From man docker-run:

       ‐‐init
          Run an init inside the container that  forwards  signals  and  reaps
       processes

[Dimokus88]

@andy108369 I think the goal is to get rid of the practice of using sleep infinity in containers. In fact, it is not needed for images, but serves mainly to help find deployment bugs.

I will reconsider the approach to using sleep infinity in the listed templates:

awesome-akash$ git grep -i sleep |grep infin
Ethereum_2.0/main.sh:sleep infinity
Falcon-7B/Dockerfile:CMD python3 falcon7b.py && sleep infinity
Sentinel-dVPN-node/main.sh:        sleep infinity
bitcoin/main.sh:sleep infinity
cryptodredge-c11/entrypoint.sh:sleep infinity
semantra/deploy.yaml:        sleep infinity ;'
softether-vpn/launch:sleep infinity

[andy108369]

@andy108369 I think the goal is to get rid of the practice of using sleep infinity in containers. In fact, it is not needed for images, but serves mainly to help find deployment bugs.

I will reconsider the approach to using sleep infinity in the listed templates:

awesome-akash$ git grep -i sleep |grep infin
Ethereum_2.0/main.sh:sleep infinity
Falcon-7B/Dockerfile:CMD python3 falcon7b.py && sleep infinity
Sentinel-dVPN-node/main.sh:        sleep infinity
bitcoin/main.sh:sleep infinity
cryptodredge-c11/entrypoint.sh:sleep infinity
semantra/deploy.yaml:        sleep infinity ;'
softether-vpn/launch:sleep infinity

@Dimokus88 yeah, that's right.
The primary process is the one that should be the main one, e.g. sshd, the app itself (or the process manager as described above in case of multi-service container). . Not sleep infinity`.

[andy108369]

Hey @Dimokus88 !
It appears that we actually can use sleep infinity or tail -f /dev/null based on my tests here

We only need to ensure to add the tini app & entrypoint code blocks as described above to the Dockerfiles using sleep infinity or tail -f /dev/null techniques as described in this issue. I.e. to make sure tini runs with PID 1 in the container app.

Here is example PR to show how I addressed the base-ssh container images:
https://github.com/akash-network/awesome-akash/pull/566/files

@Dimokus88 please proceed updating the Dockerfiles (and SDL if applicable) accordingly. Make sure to test they can actually run after updating & building them locally.