Rootless Podman Quadlet #3487
Replies: 31 comments 90 replies
-
Known issuesI've encountered some problems, however it is unknown whether these are actual bugs related to Podman or just usage errors.
For, now, you can manually update the mastercontainer by running systemctl --user stop nextcloud-aio-mastercontainer.service
podman pull docker.io/nextcloud/all-in-one:latest
systemctl --user start nextcloud-aio-mastercontainer.service |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. Will take some time until FCOS includes this. |
Beta Was this translation helpful? Give feedback.
-
|
FYI Podman 4.8.0 has been released a few days ago and is already present in the Fedora 39 repos. Should be only a matter of the CoreOS auto updater to pick that up. |
Beta Was this translation helpful? Give feedback.
-
|
@jennydaman Your quick fix doesn't work anymore as @szaimen did this commit: f935993 |
Beta Was this translation helpful? Give feedback.
-
|
@p-fruck and @loeffelpan done |
Beta Was this translation helpful? Give feedback.
-
|
Is there any workarounds for borgBackup? In fact, the database always starts with: |
Beta Was this translation helpful? Give feedback.
-
|
Will this solution also work with a Podman compose file? What is de advantage of using a systemd unit file? |
Beta Was this translation helpful? Give feedback.
-
|
On the surface level, the main advantage of Quadlet is convenient for people who like systemd and are already comfortable with its syntax. |
Beta Was this translation helpful? Give feedback.
-
|
@jennydaman: thanks for the clarification! I didn't know that it was possible to start containers (with Podman) in unit files! Seems like a clean/efficient solution! |
Beta Was this translation helpful? Give feedback.
-
|
Soft reset instructions systemctl --user stop nextcloud-aio-mastercontainer
containers=$(podman ps -a -f 'name=^nextcloud-aio' --format='{{.Names}}')
podman stop $containers
podman rm $containers
systemctl --user start nextcloud-aio-mastercontainer |
Beta Was this translation helpful? Give feedback.
-
|
Hey, I'm using podman (rootless) in favor of docker on FCOS. My first attempt to get this working like described above ended in an error at domain check. Maybe this is a networking issue as caddy is proxying to Next step was to skip domaincheck but that ended in an Bonus: |
Beta Was this translation helpful? Give feedback.
-
|
Hey there @loeffelpan,
are you running behind cloudflare? If so, this might be the reason for your domaincheck issues. When disabling the domaincheck, do the application errors look similar to the ones described in this comment? #3487 (reply in thread)
If so, this will be fixed in a newer version of podman. 4.7.0 doesn't contain this fix and I haven't tested 4.7.2 until yet, but I don't think it includes my fix already.
Until the fixed podman version is released, you can also patch the source code of your nextcloud AIO as described by @jennydaman here: #3487 (reply in thread)
|
Beta Was this translation helpful? Give feedback.
-
|
How do you turn off domaincheck? Can I use this? Environment=SKIP_DOMAIN_VALIDATION=true |
Beta Was this translation helpful? Give feedback.
-
|
@kri164 Yes. Correct. |
Beta Was this translation helpful? Give feedback.
-
|
@p-fruck You're correct. With Is the need of If I an correct this should be mentioned in the initial post for anyone who is joining this discussion. |
Beta Was this translation helpful? Give feedback.
-
|
Interestingly, my local setup is created with the network in bridge mode by default (podman 4.7.0). Might have to do with my system default (Fedora CoreOS 39 beta), though I did not check any further. @loeffelpan what exactly would you like to mention? Afaict the default network mode of podman might deviate depending on how the given linux distribution packages podman, so I would simply mention that bridge network must be used (as already done by @jennydaman ) |
Beta Was this translation helpful? Give feedback.
-
|
I think this should not differ between distros. I'm also on podman 4.7.0 with FCOS 38. I was pointing to this one from the initial post.
There is Allover, I think @jennydaman should modify this as simply needed and not a workaround. Regardless of distro/package differences or rootful/rootless setups. |
Beta Was this translation helpful? Give feedback.
-
|
podman -v podman logs nextcloud-aio-mastercontainer |
Beta Was this translation helpful? Give feedback.
-
|
This error seems like its the same I mentioned above, related to podman. |
Beta Was this translation helpful? Give feedback.
-
|
I'm confused that this question has not beeing asked. Maybe I missed something. |
Beta Was this translation helpful? Give feedback.
-
|
Thats quite a good idea. Thanks! @szaimen |
Beta Was this translation helpful? Give feedback.
-
|
@loeffelpan and @szaimen thanks for bringing |
Beta Was this translation helpful? Give feedback.
-
|
In my case (on FCOS) |
Beta Was this translation helpful? Give feedback.
-
|
Can confirm that its disabled by default. This should not be anything CoreOS specific though. You can enable it in the ignition file for new installations, but a simple |
Beta Was this translation helpful? Give feedback.
-
|
I did this. But for some reason (maybe zincati update finaliziation?) its disabled again. Any idea? |
Beta Was this translation helpful? Give feedback.
-
|
Anyone using collabora in this setup? Without collabora nextcloud is working fine. |
Beta Was this translation helpful? Give feedback.
-
|
Not using collabora yet but I had some DNS issues trying to access my IdP when configuring OIDC for SSO. Can you try to |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
It depends. For me curl was working but the php request didn't until I executed nslookup for the domain once. Apparently the dns entry had to be cached first for some odd reason |
Beta Was this translation helpful? Give feedback.
-
|
To solve this I started another discussion which brought me on the right track. So running collabora in our rootless setup isn't possible right now - regardless docker or podman. |
Beta Was this translation helpful? Give feedback.
-
|
I followed the above steps on Fedora Silverblue 39 and was unable to start the |
Beta Was this translation helpful? Give feedback.
-
|
no dice for me, unfortunately |
Beta Was this translation helpful? Give feedback.
-
|
Is SELinux enabled on Silverblue? As it's basically Fedora, it should. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe this helps? https://github.com/nextcloud/all-in-one#are-there-known-problems-when-selinux-is-enabled |
Beta Was this translation helpful? Give feedback.
-
|
Indeed that would help. But disabling security is not my way to go. I'm looking for a way to allow socket access by rule for SELinux instead. |
Beta Was this translation helpful? Give feedback.
-
|
Honestly, I feel like running with SELinux in enforcing mode and disabling security opts for the master container only might be the way to go. If you want to build your own SELinux policy, maybe this reply can help ;) |
Beta Was this translation helpful? Give feedback.
-
|
Some notes from my last install. Hope it helps. cgroupsRunning containers as rootless requires cgroups version 2. Version 1 or hybrid mode v1/v2 doesnt work. Check the status SELinuxCheck the status with command NetavarkTodays versions of Podman comes with Netavark module, but existing installations are configured for CNI. If you have problems with intercontainer DNS resolution, switch it. Backup restoreBackup restore doesnt work during initial installation. Error is "Permission denied". Not sure if common bug or rootless install only. As a workaround, you can symlink backup directory to /tmp and use it for import |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for these instructions. I was able to get this working through cloudflare tunnel by adding a "Bypass - include everyone" rule for the zero trust app. (Allow - include everyone did not work because it continued to ask for cloudflare OTP, which broke the mobile apps with "Malformed server configuration"). I will need to make modifications to config.php, namely |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I don't think so but you could use some kind of script/init-container which executes the |
Beta Was this translation helpful? Give feedback.
-
|
For future reference, I solved this (from the host OS) by editing the file at |
Beta Was this translation helpful? Give feedback.
-
|
Fedora 39 here. Any ideas? |
Beta Was this translation helpful? Give feedback.
-
|
Maybe this one help https://github.com/dpw/selinux-dockersock |
Beta Was this translation helpful? Give feedback.
-
|
In case you do not want to deal with SELinux policies, you might want disable SELinux labeling for socket as explained in the docs |
Beta Was this translation helpful? Give feedback.
-
|
@p-fruck That seems to work. |
Beta Was this translation helpful? Give feedback.
-
|
Any way to create a policy on Fedora Core OS? I can't figure it out. |
Beta Was this translation helpful? Give feedback.
-
|
I think this question should be asked in the podman discussions. There official answer seems to be disabling SELinux labels for the specified container though. Feel free to link to link your discussion here ;) |
Beta Was this translation helpful? Give feedback.
-
|
On any mastercontainer update I got this error (in the logs of watchtower container): Same on starting newly created mastercontainer. |
Beta Was this translation helpful? Give feedback.
-
|
Am I the only one with problems on updates (of mastercontainer or any other container of aio)? |
Beta Was this translation helpful? Give feedback.
-
|
for me, at the moment, automated updates always fail to restart containers, and occasionally crash the ncaio container as well. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah. Containers can be created but not started. Seems due to the above error concerning memory swapiness. Maybe someone can take a look? |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure that I have the same issue as you, although the symptoms are similar (ncaio crashing or not starting containers) here are my latest watchtower logs. note the date is feb 2 although just this morning (feb 4) I experienced aio stopping all containers and failing to start them If the problem is docker.io limits, I can temporarily solve by authenticating, but it seems like something deeper is going on here |
Beta Was this translation helpful? Give feedback.
-
|
You're right. Seems to be not the same error. |
Beta Was this translation helpful? Give feedback.
This comment has been hidden.
This comment has been hidden.
-
|
Hey! It seems that for some reason podman needs catatonit in order to create other containers. I didn't know that it could have saved a couple of days of troubleshooting. I'm just documenting it here in case that someone has the same problem! |
Beta Was this translation helpful? Give feedback.
-
|
I now found out there's an issue with podman-restart. For example: |
Beta Was this translation helpful? Give feedback.
-
|
I think it's not so good to use podman-restart for restarting the containers after a reboot. To the Then create the (don't forget to chmod +x the file) This way systemd will start the master container and make sure that it's restarted due to |
Beta Was this translation helpful? Give feedback.
-
|
Trying this on Arch Linux arm on RPi 5 (it may or may not be the case) is and I get the following: When connected to localhost:11001: Slim Application Error. Then after page refresh: Domaincheck container is not running. In the journal: Also on aardvark-dns is installed. |
Beta Was this translation helpful? Give feedback.
-
|
Resolved. Now this: |
Beta Was this translation helpful? Give feedback.
-
|
whats in your |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Resolved by stopping/disabling non-user podman.service and podman.socket, and by using |
Beta Was this translation helpful? Give feedback.
-
|
How are you guys running caddy with this setup ? Are you running it with podman-rootless or with podman-root? |
Beta Was this translation helpful? Give feedback.
-
|
I'm using nginx as the reverse proxy and ssl offloader. |
Beta Was this translation helpful? Give feedback.
-
|
i had success by adding npm proxy environment variables. nextcloud-aio-mastercontainer.containerIn npm you need to adjust
Screenshot
If you can't login after installation check if your clients using old credentials. i hope this helps |
Beta Was this translation helpful? Give feedback.
-
|
For some reason, |
Beta Was this translation helpful? Give feedback.
-
|
Resolved by adding |
Beta Was this translation helpful? Give feedback.
-
|
It would appear that this is not propagated to the child containers. Any ideas on how to change DNS for all the Nextcloud containers? Edit: Ah, no matter, this is resolved by changing DNS for the machine and/or router. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
That seems unrelated and more like a corrupt disk |
Beta Was this translation helpful? Give feedback.
-
|
Do I have to use a reverse proxy? Or can I just publish the container to 8443 directly by changing to ... as the Nextcloud AIO documentation defaults to doing? |
Beta Was this translation helpful? Give feedback.
-
|
After a server reboot, I could not get the AIO container up with: The permissions on No idea why a reboot changed that... wasted 4 hours on this... |
Beta Was this translation helpful? Give feedback.
-
|
Is there anything useful in If nothing helps, does |
Beta Was this translation helpful? Give feedback.
-
running the chmod does fix this. What I don't know is why it changes. |
Beta Was this translation helpful? Give feedback.
-
|
After updating the mastercontainer, I encountered the same socket permission error. podman.sock was 660 Adding Cannot connect to the docker socket. Cannot proceed.
nextcloud-aio-mastercontainer[1957555]: Did you maybe remove group read permissions for the docker socket? AIO needs them in order to access the docker socket.
Suggestions for further investigation? |
Beta Was this translation helpful? Give feedback.
This comment has been hidden.
This comment has been hidden.
-
|
I've AIO working, but every time that I reboot my server, the nextcloud containers do not start, even if AIO is up&&running. Is there a way to auto run the "child" containers without the need to access AIO page and press the button? |
Beta Was this translation helpful? Give feedback.
-
|
seems to need to write custom quadlets in podman to get this up&&running after each reboot |
Beta Was this translation helpful? Give feedback.
-
|
I got stuck on the first step. I can't even install it successfully. than i got error: |
Beta Was this translation helpful? Give feedback.
-
|
Re-read the instructions. Specifically: Change /run/user/1001/podman/podman.sock to the path of your Podman socket (podman info --format '{{ .Host.RemoteSocket.Path }}') There is no |
Beta Was this translation helpful? Give feedback.
-
|
You actually can do this if you enable to podman socket as root and having the podman-docker package installed or just point directly to the podman socket. The problem seems to be that the socket doesn't work with SELinux enabled, even with passing |
Beta Was this translation helpful? Give feedback.
-
|
Here's my currently working example: |
Beta Was this translation helpful? Give feedback.
-
It's working for me |
Beta Was this translation helpful? Give feedback.
-
|
For context, I believe this is similar to why you also need to disable SELinux when you're doing podman in Kubernetes as well. I realize that it isn't ideal for everyone, but it is a reality for some similar podman deployment operations and not only a Nextcloud AIO issue. (In this case, you do not need to disable SELinux on the host, but just disable it in the specific podman run!)
|
Beta Was this translation helpful? Give feedback.
-
|
Hello! I'm attempting to run Nextcloud in this fashion but I have some concerns. For context, my environment is is using Podman Quadlets on a Fedora VM. Users and permissions are managed using FreeIPA. All persistent volumes are stored on NFS shares that are also Kerberized. I am running this as a specific user using the above configurations (although I am heavily considering the manual installation in a Pod to allow this to run rootless under root). The reason I bring this up is that I can't actually start the Nextcloud container itself as it can't chown the data directory to Note:
|
Beta Was this translation helpful? Give feedback.
-
|
Ok I think I just need to map UID 33 into the container, but I'm figuring out how to get NFS to export UID 33. |
Beta Was this translation helpful? Give feedback.
-
|
Ref. https://www.redhat.com/sysadmin/rootless-podman-nfs but you have an extra complexity with sssd. |
Beta Was this translation helpful? Give feedback.
-
|
Also, this discussion: containers/podman#16018 |
Beta Was this translation helpful? Give feedback.
-
|
I created a tool to manage podman and quadlets from repositories, and with it this installation can be run like so: The tool is using git repositories with folders containing all the quadlet files. The default the tool works with is https://github.com/rgolangh/podman-quadlets , but you can use your own of course. If you're interested please check-out https://github.com/rgolangh/pq - PRs or issues/discussions and feedback are welcomed. |
Beta Was this translation helpful? Give feedback.
-
|
This is cool - it seems like a sort of helm chart way to deploy things with podman. Do you also have a way to update the existing deployment with newer image refs? |
Beta Was this translation helpful? Give feedback.
-
|
the beauty is that podman does that for you using podman auto-update.service and to opt-in just add this to the quadlet file: This should result in a label on the container podman creates to be picked up by auto-update. |
Beta Was this translation helpful? Give feedback.
-
|
Excellent work! I love that you also have a repo ref for Valkey as well! |
Beta Was this translation helpful? Give feedback.
-
|
When going into https://127.0.0.1:8443 I get an |
Beta Was this translation helpful? Give feedback.
-
|
Can you post your caddyfile? |
Beta Was this translation helpful? Give feedback.
-
|
After updating to version 30.0.2, the AppAPI feature is flooding my logs with errors related to accessing the docker.sock. Suggestions? Disable AppAPI? Point it in the right direction somehow? |
Beta Was this translation helpful? Give feedback.
-
|
Install podman-docker and |
Beta Was this translation helpful? Give feedback.
-
|
Both were already installed prior to this update. Are you suggesting podman.socket needs to be enabled for the root user? podman.socket is enabled as the podman user. My intent is to use podman as a rootless user. |
Beta Was this translation helpful? Give feedback.
-
Do you have DOCKER_HOST set? and in the unit file of nextcloud: |
Beta Was this translation helpful? Give feedback.
-
|
It is worth making sure podman.socket is active as the user |
Beta Was this translation helpful? Give feedback.
and others
-
This guide sets up Nextcloud-AIO using Podman in rootless mode and Quadlet behind a reverse proxy.
0. Install Podman
Podman version 4.8.0 or above is required. Older versions of Podman require workarounds, see edit history and discussion below.
1. Set up a reverse proxy
Consult the upstream documentation on how to do this: https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
An example
Caddyfilemight look like this:2. Create Systemd Unit File
Create the file
~/.config/containers/systemd/nextcloud-aio-mastercontainer.containerwith the following content:Important notes:
/run/user/1001/podman/podman.sockto the path of your Podman socket (podman info --format '{{ .Host.RemoteSocket.Path }}')nextcloud-aio. The optionNetwork=bridgeenables this behavior, for details see podman network connect not implemented for slirp4netns containers/podman#19577 (comment)Also, create the file
~/.config/containers/systemd/nextcloud-aio-mastercontainer.volume:3. Start the Services
4. (Optional) Configure Containers to Restart After Reboot
Podman is daemonless so unlike Docker, containers do not restart automatically after reboot.
To enable restart of Podman containers after reboot, see containers/podman#20418 (comment)
5. Business As Usual
Go to https://cloud.example.com:8443 to access the Nextcloud AIO interface and start the Nextcloud server.
Notes
Backups and mastercontainer self-updating might not work, these details have yet to been sorted out.
Beta Was this translation helpful? Give feedback.
All reactions