-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rootless containers cannot be started: RLIMIT_NPROC error #1507
Comments
This could be containers/podman#18696 (fixed by containers/podman#18721 which isn't in any release yet). |
I think so too. @hrismarin : To confirm you can run |
15282 vs 15257, so it should be the same issue. |
What I don't quite fully understand is that the Was it a different piece of software that caused another change that affected containers? |
Here should be the full list of changes:
|
podman isn't really broken in either version. It's just that you can not start a container that was created in 38.20230514.1.0 in 38.20230527.1.1. Upon starting the container the runtime will set the container's resource limits (number of processes, files, etc). A regular user (as is the case here) is only allowed to set the resource limits to the current hard limit at most. Trying to set a soft limit higher than the hard limit will results in the error seen here (or trying to increase the hard limit). https://linux.die.net/man/2/setrlimit The problem with podman prior to the fix is that the default resource limits for a container were set at container creation time instead of when it's started. So if for any reason the resource limits for an unprivileged user decrease between the container's creation time and when it's launched, this error will result. In this case podman (actually the runtime) will still try to set the higher resource limit that was saved in the containers config at creation time. This will fail as the current hard limit is lower.
I don't know what package caused the decrease in the resource limits. Maybe linux-firmware? It's also affected by the amount of memory in the system. So people can potentially run into this issue without doing any updates. Once the fix is in an upstream release and it makes it into coreos, people can still run into this issue unfortunately, because containers that were created with a version of podman without the fix will still have the resource limit set from creation time. So a future coreos update or memory configuration change even after the upstream fix is in coreos can still trigger this issue on older containers. |
Unfortunately
did not resolve the issue. |
Thank you @travier! The only reason for me to start previously created containers, so far, is that they are automatically started by their systemd services. This is perhaps not the most proper and recommended way. I have already tried and implemented some workarounds. I'm just trying to participate as much as I can to prevent similar issues in the future... if that's even possible. |
Going off of #1507 (comment) containers/podman@249f046 is in |
The fix for this went into |
The fix for this went into |
Since I no longer run containers without the
However, in the following scenario:
The container is running, so it looks like the fix in podman is working. Please let me know if you need more details or additional tests. I would be glad to do so. |
Containers that have been created from older (unfixed) podman versions can not be "fixed" after the fact. This only fixes it for new containers created by podman 4.6. |
Yes, it was already assumed by @Cydox:
I just wanted to check. |
WorkaroundsThe issue is fixed in podman 4.6 but containers created before this release can not be fixed without being re-created or raising the Save and restore containersYou can export and import your existing containers. See for example:
Set a higher ulimit for your userWrite the following file, replacing
Reboot to apply the configuration change. |
The fix for this went into |
This is still broken in latest Kinoite/Podman.
|
@tidux What podman version was the container created with? It is only fixed for containers that were created with 4.6 or later. |
I ended up deleting the containers with |
Describe the bug
I am running several WordPress containers behind nginx reverse-proxy in a KVM/QEMU VM. I am also running a pod, which includes mariadb and a Document/File Sharing platform on a bare metal machine.
Recently, when FCOS was updated to 38.20230527.1.1 next, after a reboot, the containers were not automatically started by their systemd services. When I try to run them manually I get the following error:
Error: unable to start container "CONTAINER ID": crun: setrlimit RLIMIT_NPROC: Operation not permitted: OCI permission denied
When I rollback to to the previous deployment containers started as usual.
Reproduction steps
podman container create
and/orpodman container run
commands must be executed without the--rm
option.Expected behavior
The container and application are up and running.
Actual behavior
podman container start CONTAINER NAME
failed withError: unable to start container "CONTAINER ID": crun: setrlimit
RLIMIT_NPROC
: Operation not permitted: OCI permission deniedSystem details
Systems 1 & 2 (Identical, Bare Metal)
Architecture: x86_64
Motherboard: Gigabyte Technology G41M-ES2L
CPU: Pentium(R) Dual-Core CPU E6300 @ 2.80GHz
RAM: 4 GB
System 3 (KVM/QEMU)
Architecture: x86_64
Host: CentOS Stream 8
vCPUs: 2
vRAM: 4 GB
System 4 (KVM/QEMU)
Architecture: x86_64
Host: Fedora Rawhide
vCPUs: 2
vRAM: 4 GB
Butane or Ignition config
Additional information
### Creating, running, starting and stopping new container from an existing image works as expected on the updated 38.20230527.1.1 next system.
Replacing kernel with older 6.2.13-300.fc38.x86_64 and also overriding crun with the older 1.8.4-1.fc38.x86_64 version on 38.20230527.1.1 next system did not solve the issue.
Systems 1 and 3 already had FCOS 38.20230514.1.0 next installed and working.
On systems 2 and 4 I repeated the reproduction steps intentionally, just in case.
The text was updated successfully, but these errors were encountered: