podman rootless startup: missing dependency in systemd PATH

[LucaFulchir]

Describe the bug

I'm on nixos unstable

I'm trying to start a rootless podman container via podman. basically define a podman container the nix way, but starting it from a user.

Podman can not start because it needs the newuidmap binary (from the shadow package, which is not in the PATH of the systemd:

Error: command required for rootless mode with multiple IDs: exec: "newuidmap": executable file not found in $PATH

Note: all works if we start the same container with the root user

Steps To Reproduce

Steps to reproduce the behavior:

1. have a non-root user
2. in nix, define a set of subuid/subgid for that user:

    users.users = {
      my-podman-user = {
        subUidRanges = [{
          startUid = 200000;
          count = 100000;
        }];
        subGidRanges = [{
          startGid = 200000;
          count = 100000;
        }];
      };

3. define a podman container (the only important thing is probably the uidmap/gidmap):

  virtualisation = {
    podman.enable = true;
    oci-containers = {
      backend = "podman";
      containers = {
        "alpine" = {
          image = "docker.io/library/alpine:latest";
          extraOptions = [
            "--name=alpine"
            "--systemd=always"
            "-h=test.fqdn"
            "--dns=1.1.1.1"
            "--network=host"
            "--uidmap=0:1:65536"
            "--gidmap=0:1:65536"
          ];
          cmd = [ "sleep" "600" ];
        };
      };
    };
  };

4. override the systemd unit of that container so it starts as a non-root user, then add a couple of environment varialbes

  systemd = {
    services = {
     "podman-alpine" = {
       serviceConfig = {
         User="my-podman-user";
         Group="my-podman-user";
       };
       environment = {
         HOME="/home/my-podman-user";
         XDG_RUNTIME_DIR="/run/user/${MY_PODMAN_USER_UID}";
         DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/${MY_PODMAN_USER_UID}/bus";
       };
     };

5. LOGIN as that user. Important: we need a valid systemd session, with a valid dbus socket. This manual login can be avoided with more nix conf (see later) but it's simpler this way
6. nix-rebuild switch
7. journactl -eu podman-alpine

Error: command required for rootless mode with multiple IDs: exec: "newuidmap": executable file not found in $PATH

Expected behavior

The podman container starts under the given user.

Additional context

This part is only for those that want to try with a more stable setup, and can be ignored as long as you login as your podman user before starting the podman container

Adapted from here: https://serverfault.com/questions/892465/starting-systemd-services-sharing-a-session-d-bus-on-headless-system

Basically we need to create a valid systemd user session for your podman user, and a dbus one, too

I did not create the systemd-user files like the guide above, but relied on system-level ones since the nix configuration only generates units on system level, not at user level AFAIK

• add systemd lingering for the podman user:

  system.activationScripts = {
    enableLingering = ''
      # remove all existing lingering users
      rm -r /var/lib/systemd/linger
      mkdir /var/lib/systemd/linger
      # enable for the subset of declared users
      touch /var/lib/systemd/linger/my-podman-user
    '';
  };

• make sure the user dbus session starts at boot (change uid accordingly):

  systemd = {
    services = {
     "dbus-podman" = {
        description = "D-Bus for podman user";
        serviceConfig = {
          User="my-podman-user";
          Group="my-podman-user";
        };
        requires = [ "dbus-podman.socket" ];
        serviceConfig = {
          "ExecStart" = "/run/current-system/sw/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation";
          "ExecReload" = "/run/current-system/sw/bin/dbus-send --print-reply --session --type=method_call --dest=org.freedesktop.DBus / org.freedesktop.DBus.ReloadConfig";
        };
       environment = {
         HOME="/home/pm-ctr";
         XDG_RUNTIME_DIR="/run/user/${MY_PODMAN_UID}";
         DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/${MY_PODMAN_UID}/bus";
       };
     };
    sockets = {
      "dbus-podman" = {
        description = "D-Bus Socket for podman";
        after = [ "network.target" ];
        socketConfig = {
          "ListenStream" = "/run/user/${MY_PODMAN_UID}/bus";
          "SocketUser" = "my-podman-user";
          "SocketGroup" = "my-podman-user";
          "SocketMode" = "0660";
        };
        wantedBy = [ "sockets.target" ];
      };
    };
  };

• add the dbus dependency in your previous container systemd override:

requires = [ "dbus-podman.service" ];

Notify maintainers

@adisbladis
@saschagrunert
@vdemeester
@zowoq

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 5.10.52, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.15`
 - channels(root): `"nixos-21.11pre316472.47eaf6727bd"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs`

[binarycode]

I'm having the same issue with rootless podman containers launched by systemd. Following configuration snippet could be used for a simple repro:

{ pkgs, ... }: let
  user = "test";
  group = "test";
  container = "rootless-podman-test";
  service = {
    serviceConfig = {
      User = user;
      Group = group;
      Type = "oneshot";
    };
    script = "${pkgs.podman}/bin/podman run --name ${container} --replace --rm docker.io/library/hello-world:latest";
    preStop = "${pkgs.podman}/bin/podman stop ${container}";
  };
in {
  systemd.services = {
    rootless-podman-test1 = service;
    rootless-podman-test2 = service // {
      path = [ pkgs.shadow ];
    };
  };
  users = {
    groups.${group} = {};
    users.${user} = {
      isNormalUser = true;
      group = group;
    };
  };
}

# loginctl enable-linger test

# systemctl start rootless-podman-test1
Job for rootless-podman-test1.service failed because the control process exited with error code.
See "systemctl status rootless-podman-test1.service" and "journalctl -xeu rootless-podman-test1.service" for details.

# journalctl -u rootless-podman-test1
Oct 20 12:26:55 oat systemd[1]: Starting rootless-podman-test1.service...
Oct 20 12:26:55 oat rootless-podman-test1-start[2539262]: Error: command required for rootless mode with multiple IDs: exec: "newuidmap": executable file not found in $PATH
Oct 20 12:26:55 oat systemd[1]: rootless-podman-test1.service: Main process exited, code=exited, status=125/n/a
Oct 20 12:26:55 oat systemd[1]: rootless-podman-test1.service: Failed with result 'exit-code'.
Oct 20 12:26:55 oat systemd[1]: Failed to start rootless-podman-test1.service.

# systemctl start rootless-podman-test2
Job for rootless-podman-test2.service failed because the control process exited with error code.
See "systemctl status rootless-podman-test2.service" and "journalctl -xeu rootless-podman-test2.service" for details.

# journalctl -u rootless-podman-test2
Oct 20 12:28:02 oat systemd[1]: Starting rootless-podman-test2.service...
Oct 20 12:28:02 oat rootless-podman-test2-start[2541461]: Error: cannot setup namespace using newuidmap: exit status 1
Oct 20 12:28:02 oat systemd[1]: rootless-podman-test2.service: Main process exited, code=exited, status=125/n/a
Oct 20 12:28:02 oat systemd[1]: rootless-podman-test2.service: Failed with result 'exit-code'.
Oct 20 12:28:02 oat systemd[1]: Failed to start rootless-podman-test2.service.

So if the shadow package is not in path of the service, the service fails with "newuidmap": executable file not found in $PATH error, but if I add it to the path manually, it fails with cannot setup namespace using newuidmap: exit status 1 error.

If I try to launch same container manually, it works fine:

# su test -c 'podman run --rm docker.io/library/hello-world:latest'
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob 2db29710123e done
Copying config feb5d9fea6 done
Writing manifest to image destination
Storing signatures

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

[binarycode]

Ok, looks like the issue was that the newuidmap binary from shadow package lacks suid bit, and a wrapper located in /run/wrappers/bin/newuidmap should be used instead.

I'm able to successfully run rootless containers using following configuration:

{ pkgs, ... }: let
  user = "test";
  group = "test";
  container = "rootless-podman-test";
in {
  systemd.services.rootless-podman-test = {
    path = [ "/run/wrappers" ];
    serviceConfig = {
      User = user;
      Group = group;
      Type = "oneshot";
    };
    script = "${pkgs.podman}/bin/podman run --name ${container} --replace --rm docker.io/library/hello-world:latest";
    preStop = "${pkgs.podman}/bin/podman stop ${container}";
  };
  users = {
    groups.${group} = {};
    users.${user} = {
      isNormalUser = true;
      group = group;
    };
  };
}

[stale]

I marked this as stale due to inactivity. → More info

[hello-smile6]

Any info on this?

[jhvst]

@binarycode's solution works on file systems which support xattrs. If you are on a filesystem which does not support xattrs, such as tmpfs (e.g., you are network booting) and your rootfs is formatted as such, you will receive a dubious error like in this StackOverflow question:

Error: writing blob: adding layer with blob "sha256:de63ba066b7c0c23e2434efebcda7800d50d60f33803af9c500f75a69fb76ffa": Error processing tar file(exit status 1): operation not supported

To fix this, follow the StackOverflow answer, i.e., add --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs" to your container parameters, e.g.:

{ pkgs, ... }: let
  user = "test";
  group = "test";
  container = "rootless-podman-test";
in {
  systemd.services.rootless-podman-test = {
    path = [ "/run/wrappers" ];
    serviceConfig = {
      User = user;
      Group = group;
      Type = "oneshot";
    };
    script = ''${pkgs.podman}/bin/podman \
      --storage-opt "overlay.mount_program=${pkgs.fuse-overlayfs}/bin/fuse-overlayfs" run \
      --name ${container} --replace --rm docker.io/library/hello-world:latest
    '';
    preStop = "${pkgs.podman}/bin/podman stop ${container}";
  };
  users = {
    groups.${group} = {};
    users.${user} = {
      isNormalUser = true;
      group = group;
    };
  };
}

Another option, as mentioned in the StackOverflow answer, is to format your filesystem, or part of it.

Mentioning this in case someone wants to write a PR on the original topic, as the same PR should likely address this edge-case as well.

[bufferUnderrun]

podman and systemd --user is a f*%?ing mess !!

i'm trying to do a simple thing : running podman container with systemd user unit, but nothing work :(

open to suggestions...

[lriesebos]

same issue here in a nix shell. podman is basically useless in its current form (nixpkgs 22.11, but unstable seems to be unchanged).

$ cat shell.nix 
{ pkgs ? import <nixpkgs> { } }:

pkgs.mkShell {
  buildInputs = with pkgs; [ shadow podman ];
}
$ nix-shell --pure

[nix-shell:~]$ podman run -it alpine:latest
ERRO[0000] running `/nix/store/50vlvr7ic48qjlb2haylsvzld76lva3a-shadow-4.11.1/bin/newuidmap 24222 0 1000 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted 
Error: cannot set up namespace using "/nix/store/50vlvr7ic48qjlb2haylsvzld76lva3a-shadow-4.11.1/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1

also open to suggestions

edit: error was produced on Ubuntu 20.04, seems to work fine for Fedora 37

[maxbrunet]

This issue seems to have been resolved in the latest release (23.05), probably by #215588

[staticdev]

@maxbrunet is this already released?

I also just installed in my fresh Debian with: nix-env -iA nixpkgs.podman and have the same issue.

I have nix (Nix) 2.16.1 and it installed podman version 4.5.1.

[maxbrunet]

Podman cannot be used with the package only, configuration around it is required:
https://search.nixos.org/options?channel=23.05&show=virtualisation.podman.enable

This issue is for NixOS I believe. On Debian, you would need to write a module yourself or use a Debian package (.deb) instead.

From the module, you would need to install it this way:

(pkgs.podman.override {
  extraPackages = [
    # setuid shadow
    "/run/wrappers"
  ]
})

Not sure what is the minimum config to get it running and if /run/wrappers is what it is called in Debian, but that's the solution to this issue.

[staticdev]

Question out of being a noob in nixpkgs, does someone know a way to run this override from @maxbrunet in imperative nix-env?

[barrelful]

Podman cannot be used with the package only, configuration around it is required: https://search.nixos.org/options?channel=23.05&show=virtualisation.podman.enable

This issue is for NixOS I believe. On Debian, you would need to write a module yourself or use a Debian package (.deb) instead.

From the module, you would need to install it this way:

(pkgs.podman.override {
  extraPackages = [
    # setuid shadow
    "/run/wrappers"
  ]
})

Not sure what is the minimum config to get it running and if /run/wrappers is what it is called in Debian, but that's the solution to this issue.

This is sad, I am also using nix package manager on Debian.. but I do not have knowledge to write a module, from what I understood of nix package manager it should be able to have packages working independent of OS, in really I do not see this happening in the example of this issue. Btw I also tried on Mac M1 ARM, same issue on MacOS, this is a pain.

It also noticed there is no word from the maintainers on this issue..
@adisbladis @saschagrunert @vdemeester @zowoq do you think you could help us?