A bug with permissions running unit from unprivileged user

[Izorkin]

Steps to reproduce bug.

Systemd configuration:

[Unit]
After=network.target
Description=Unit App Server

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/xlc05n9qjwym4zhgx5251h7fj5z4ivy7-glibc-locales-2.27/lib/locale/locale-archive"
Environment="PATH=/nix/store/w8rlmb95jyz8qjgm6qzjg6qdlf6qqkda-curl-7.67.0-bin/bin:/nix/store/jc34mzphlwyg4p5w4d0lix2cj3yppqwd-coreutils-8.31/bin:/nix/store/z39bagbc4k5xa76cry58lrk0lnxay5r1-findutils-4.7.0/bin:/nix/store/33szav6q4ifp10brnkn9f4shigzcc256-gnugrep-3.3/bin:/nix/store/6j98mvsnqgj6bzrasd96s8b134z8iz34-gnused-4.7/bin:/nix/store/337hhdvk6d8d5x01lrs6b21h0lrvc6z6-systemd-243.3/bin:/nix/store/w8rlmb95jyz8qjgm6qzjg6qdlf6qqkda-curl-7.67.0-bin/sbin:/nix/store/jc34mzphlwyg4p5w4d0lix2cj3yppqwd-coreutils-8.31/sbin:/nix/store/z39bagbc4k5xa76cry58lrk0lnxay5r1-findutils-4.7.0/sbin:/nix/store/33szav6q4ifp10brnkn9f4shigzcc256-gnugrep-3.3/sbin:/nix/store/6j98mvsnqgj6bzrasd96s8b134z8iz34-gnused-4.7/sbin:/nix/store/337hhdvk6d8d5x01lrs6b21h0lrvc6z6-systemd-243.3/sbin"
Environment="TZDIR=/nix/store/imqrggn4dqdsdhzsv679wsg1vbhil9zr-tzdata-2019c/share/zoneinfo"

ExecStart=/nix/store/1b6rws6mcs26463xiqs80mrk9c6ijpn2-unit-1.13.0/bin/unitd --control 'unix:/run/unit/control.unit.sock' --pid '/run/unit/unit.pid' --log '/var/log/unit/unit.log' --state '/var/spool/unit' --no-daemon --user unit --group unit
ExecStartPost=/nix/store/7i6gg9knwhm9ixs1vr49ivhgc3iwamxw-unit-script-unit-post-start
ExecStartPre=/nix/store/7c495pr4dw64n2d7ibgxlcmccn63vpk9-unit-script-unit-pre-start

User=unit
Group=unit

RuntimeDirectory=unit
RuntimeDirectoryMode=0750

AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID

Unit configuration:

      {
        "listeners": {
          "*:8373": {
            "application": "php_72"
          }
        },
        "applications": {
          "php_72": {
            "type": "php 7.2",
            "processes": 1,
            "user": "test-user",
            "group": "test-user",
            "root": "/var/www",
            "index": "index.php",
            "options": {
              "admin": {
                "max_execution_time": "30",
                "max_input_time": "30",
                "display_errors": "on",
                "display_startup_errors": "on",
              }
            }
          }
        }
      }

Test script:

<?php
  posix_setgid(1);
  posix_setuid(1);
  $fd = fopen('/tmp/test_file','a');
  fwrite($fd,"test_write\n");
  fclose($fd);
?>

Result:

curl localhost:8373
ls -lah /tmp/test_file

-rw-r--r-- 1 1 wheel 11 дек  9 10:21 /tmp/test_file

If run from root:

-rw-r--r-- 1 test-user test-user 11 дек  9 10:13 /tmp/test_file

cc @tiago4orion

[i4ki]

Hi @Izorkin

I ran your example here (manually, without the systemd unit). I got:

$ ls -lha /tmp/test_file 
-rw-rw-r-- 1 daemon daemon 11 Dec  9 09:58 /tmp/test_file
$ id daemon
uid=1(daemon) gid=1(daemon) groups=1(daemon)

Could you check running it manually?

$ sudo setcap cap_setuid,cap_setgid,cap_net_bind_service+ep /home/vagrant/sbin/unitd
$  /home/vagrant/sbin/unitd

[i4ki]

Also, show the output of:

$ umask
0002

It would be good if you can show the output of this in the script as well:

<?php

system("umask");

[Izorkin]

Result umask shell
022
Result umask php
0022

Error start unitd with cap_setgid,cap_setuid,cap_net_bind_service+ep
[warn] 30740#30740 Unit is running unprivileged, then it cannot use arbitrary user and group.

[i4ki]

$ umask
0002
$ touch /tmp/test1
$ ls -lha /tmp/test1
-rw-rw-r-- 1 vagrant vagrant 0 Dec  9 10:37 /tmp/test1
$ umask 0022
$ touch /tmp/test2
$ ls -lha /tmp/test2
-rw-r--r-- 1 vagrant vagrant 0 Dec  9 10:39 /tmp/test2

This is the expected behavior, right? Or maybe I misunderstand your question. What do you mean by bug permissions?

[Izorkin]

Incorrectly translated into english.
When starting the process, a unit from a regular user with the rights CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID can create a file as root.
When running as root, there is no such error.

[i4ki]

Ok, Got it.

The problem is that capabilities are copied to child processes but the inheritable set (that can be cleared) is only used for execve(2). This is strange because root has those capabilities set but they are dropped when setuid in the children, but this doesn't happen when the capabilities are set in the file.

I'll have a look if it needs to set any additional capability to file to make it work as root. If there's no way around, then we will have to drop it explicitly in Unit.

This behavior doesn't happen with external apps because they are execve'd (the inheritable set takes place).

Thanks for the report.

[i4ki]

Hi @Izorkin

Can you test the patch below?

diff -r ed17ce89119f src/nxt_capability.c
--- a/src/nxt_capability.c      Fri Dec 06 17:02:23 2019 +0000
+++ b/src/nxt_capability.c      Mon Dec 09 23:23:00 2019 +0000
@@ -93,6 +93,26 @@ nxt_capability_specific_set(nxt_task_t *
     return NXT_OK;
 }
 
+
+nxt_int_t
+nxt_capability_drop_all(nxt_task_t *task)
+{
+    struct __user_cap_header_struct hdr;
+    struct __user_cap_data_struct data[2];
+
+    hdr.version = nxt_capability_linux_get_version();
+    hdr.pid = nxt_pid;
+
+    nxt_memset(data, 0, sizeof(data));
+
+    if (nxt_slow_path(nxt_capset(&hdr, data) == -1)) {
+        nxt_alert(task, "failed to drop capabilities %E", nxt_errno);
+        return NXT_ERROR;
+    }
+
+    return NXT_OK;
+}
+
 #else
 
 static nxt_int_t
diff -r ed17ce89119f src/nxt_capability.h
--- a/src/nxt_capability.h      Fri Dec 06 17:02:23 2019 +0000
+++ b/src/nxt_capability.h      Mon Dec 09 23:23:00 2019 +0000
@@ -14,4 +14,6 @@ typedef struct {
 NXT_EXPORT nxt_int_t nxt_capability_set(nxt_task_t *task,
     nxt_capabilities_t *cap);
 
+NXT_EXPORT nxt_int_t nxt_capability_drop_all(nxt_task_t *task);
+
 #endif /* _NXT_CAPABILITY_INCLUDED_ */
diff -r ed17ce89119f src/nxt_process.c
--- a/src/nxt_process.c Fri Dec 06 17:02:23 2019 +0000
+++ b/src/nxt_process.c Mon Dec 09 23:23:00 2019 +0000
@@ -264,7 +264,7 @@ cleanup:
 static void
 nxt_process_start(nxt_task_t *task, nxt_process_t *process)
 {
-    nxt_int_t                    ret, cap_setid;
+    nxt_int_t                    ret, cap_setid, drop_caps;
     nxt_port_t                   *port, *main_port;
     nxt_thread_t                 *thread;
     nxt_runtime_t                *rt;
@@ -285,9 +285,12 @@ nxt_process_start(nxt_task_t *task, nxt_
 
     cap_setid = rt->capabilities.setid;
 
+    drop_caps = cap_setid;
+
 #if (NXT_HAVE_CLONE_NEWUSER)
-    if (!cap_setid && NXT_CLONE_USER(init->isolation.clone.flags)) {
+    if (NXT_CLONE_USER(init->isolation.clone.flags)) {
         cap_setid = 1;
+        drop_caps = 0;
     }
 #endif
 
@@ -301,6 +304,12 @@ nxt_process_start(nxt_task_t *task, nxt_
         if (nxt_slow_path(ret != NXT_OK)) {
             goto fail;
         }
+
+#if (NXT_HAVE_LINUX_CAPABILITY)
+        if (drop_caps && nxt_capability_drop_all(task) != NXT_OK) {
+            goto fail;
+        }
+#endif
     }
 
     rt->type = init->type;

Thanks.

[Izorkin]

Hi @tiago4orion this patch does not fix the error.
-rw-r--r-- 1 1 wheel 11 дек 10 09:40 /tmp/test_file

[Izorkin]

Sorry, recheked - correct work!
-rw-r--r-- 1 test-user test-user 33 дек 10 10:02 /tmp/test_file

[i4ki]

Ok. I don't like this solution because dropping all capabilities effectively prevents some use cases (applications that need to do administrative jobs, like mount/umount or privileged-network stuff, eg.: like the ones requiring CAP_SYS_ADMIN).

This Linux behavior is really odd, from man capabilities:

Effect of user ID changes on capabilities
       To preserve the traditional semantics for transitions between 0 and nonzero user IDs, the  kernel  makes  the
       following  changes  to  a thread's capability sets on changes to the thread's real, effective, saved set, and
       filesystem user IDs (using setuid(2), setresuid(2), or similar):

       1. If one or more of the real, effective or saved set user IDs was previously 0, and as a result of  the  UID
          changes  all  of  these  IDs  have  a nonzero value, then all capabilities are cleared from the permitted,
          effective, and ambient capability sets.

       2. If the effective user ID is changed from 0 to nonzero, then all capabilities are cleared from  the  effec‐
          tive set.

       3. If  the  effective user ID is changed from nonzero to 0, then the permitted set is copied to the effective
          set.

       4. If the filesystem user ID is changed from 0 to nonzero (see setfsuid(2)), then the following  capabilities
          are  cleared  from  the  effective  set:  CAP_CHOWN,  CAP_DAC_OVERRIDE,  CAP_DAC_READ_SEARCH,  CAP_FOWNER,
          CAP_FSETID, CAP_LINUX_IMMUTABLE  (since  Linux  2.6.30),  CAP_MAC_OVERRIDE,  and  CAP_MKNOD  (since  Linux
          2.6.30).   If  the  filesystem  UID  is changed from nonzero to 0, then any of these capabilities that are
          enabled in the permitted set are enabled in the effective set.

       If a thread that has a 0 value for one or more of its user IDs wants to prevent its permitted capability  set
       being  cleared  when it resets all of its user IDs to nonzero values, it can do so using the SECBIT_KEEP_CAPS
       securebits flag described below.

It only clears the capabilities after setuid if the process was previously root or any other thread uid is still root.

@VBart We can make the drop all capabilities the default, but add an option where users can explicitly choose what capabilities to add or drop in the application.

LXC has a keep and drop set in the configuration: https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html#lbAX

Docker has an --cap-add and --cap-drop: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

what do you think?

[VBart]

@tiago4orion Having two options to specify what to add and what to drop looks too complex. Also, it's hard to predict what the final capabilities list the process will eventually have. Why don't just specify the exact capabilities list and let Unit do the best to provide them? We need just one option/object similar to "namespaces".

[Izorkin]

@tiago4orion running unit on root and rechecking capability
Main process:

sudo cat /proc/22685/status | grep Cap
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000

capsh --decode=0x0000003ff7fcffff
0x0000003ff7fcffff=cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read

App process:

sudo cat /proc/22719/status | grep Cap
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000

capsh --decode=0x0000003fffffffff
0x0000003fffffffff=cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read

How to reset CapEff for app process to 0000000000000000 ?

[i4ki]

Hi @Izorkin

Some questions:

Is your app process using the credential namespace? If so, that works as intended.
If not, are you using unit unmodified or with the patch I sent here applied?

If you are using Unit without any changes, then when running unitd as root it will drop all capabilities of the app if the app's user is different than root and app is not using credential namespaces.

It would be useful if you show how you are running unit and say if the app config has anything special (like user = root).

The patch I sent here before is not applied to Unit repo yet, because it makes some use cases impossible.
I have another patch that makes Unit a "capabilities aware" application and then added options to explicitly set the ambient, permitted and effective capabilities of app processes and dropping everything by default.

I don't have an ETA for when it will be released, as it still needs some work and reviews.
I recommend you to not set ambient capabilities in systemd for now and run Unit as root, as recommended, until we make it capability aware.

[Izorkin]

Hi @i4ki

Is your app process using the credential namespace? If so, that works as intended.
If not, are you using unit unmodified or with the patch I sent here applied?

Yes, checked with namespace.
Problem checking without namespace. Need check on a clean VM.
2020/04/14 23:13:48 [alert] 6657#6657 clone() failed while creating ""php-72-roundcube" application" (28: No space left on device)

I recommend you to not set ambient capabilities in systemd for now and run Unit as root, as recommended, until we make it capability aware.

Created PR in Nixos to remove patch and run Unit as root.

It would be useful if you show how you are running unit and say if the app config has anything special (like user = root).

Latest systemd config

[Unit]
After=network-online.target
Description=Unit App Server

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/wwcid867dc01j7aklwm06kiq597g3x8v-glibc-locales-2.30/lib/locale/locale-archive"
Environment="PATH=/nix/store/fs0h13zxb5hcv1vr6yf2ijzp943jb6hy-coreutils-8.31/bin:/nix/store/fam612p31mimv4z47sjysxd3cgbzahaj-findutils-4.7.0/bin:/nix/store/i7gpf3lqp3wwc60snbl1b0m69g3pis4f-gnugrep-3.4/bin:/nix/store/w223x85ixncjxs0bqp9z1dfx64b16hb0-gnused-4.8/bin:/nix/store/mc5vnqnvwh7falbapfs8b5wq1z4fwqab-systemd-243.7/bin:/nix/store/fs0h13zxb5hcv1vr6yf2ijzp943jb6hy-coreutils-8.31/sbin:/nix/store/fam612p31mimv4z47sjysxd3cgbzahaj-findutils-4.7.0/sbin:/nix/store/i7gpf3lqp3wwc60snbl1b0m69g3pis4f-gnugrep-3.4/sbin:/nix/store/w223x85ixncjxs0bqp9z1dfx64b16hb0-gnused-4.8/sbin:/nix/store/mc5vnqnvwh7falbapfs8b5wq1z4fwqab-systemd-243.7/sbin"
Environment="TZDIR=/nix/store/iyhn7j4g2yn5r9krygapn58pwsk0yqd5-tzdata-2019c/share/zoneinfo"

ExecStart=/nix/store/6mfq2pf7g7rkjq4ggxwcla65l5pd3340-unit-1.16.0/bin/unitd --control 'unix:/run/unit/control.unit.sock' --pid '/run/unit/unit.pid' \
                         --log '/var/log/unit/unit.log' --state '/var/spool/unit' \
                         --user unit --group unit

ExecStartPost=/nix/store/00j108jypk79qrhjc720ha4xb85q34pq-unit-script-unit-post-start
ExecStartPre=/nix/store/bdai2l1mwks6fqdb328cjgvjxcpin4df-unit-script-unit-pre-start
ExecStop=/nix/store/lrlpjnp93x4ajakydq1rs982b90h8dwc-curl-7.69.1-bin/bin/curl -X DELETE --unix-socket '/run/unit/control.unit.sock' 'http://localhost/config'

LockPersonality=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PIDFile=/run/unit/unit.pid
PrivateDevices=true
PrivateMounts=true
PrivateTmp=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=strict
ReadWritePaths=/var/spool/unit
ReadWritePaths=/var/log/unit
ReadWritePaths=/var/data/mail
RestrictRealtime=true
RuntimeDirectory=unit
RuntimeDirectoryMode=0750
Type=forking

Next error.
If run unit as non-root and this kernel patches - anthraxx/linux-hardened@104f440
error start unit

2020/04/14 12:16:58 [alert] 3819#3819 clone() failed while creating ""php-72-roundcube" application" (1: Operation not permitted)

Need unprivileged CLONE_NEWUSER ?
From root normal starting.

[i4ki]

Hi @Izorkin

If running unitd as root, you should have no problem to use namespaces. If you do receive "permission denied" even if running as root, then maybe the problem is your NoNewPrivileges=true. I never tested it on the parent process when it's already privileged and the man page says nothing about namespaces, but maybe that's the cause. I will test this soon.

About your other error related to "no space left on device", I recommend you to read the man page for the clone syscall and read the various cases where ENOSPC can return. It's most likely that you have some hardning in place that limits the number of nested namespaces in the system.

[Izorkin]

Hi @i4ki
Sorry, incorrect checking...
Unit correct working.

anthraxx linux hardened + namespcase

main process
CapInh: 0000000000000000
CapPrm: 0000003ff7fcffff
CapEff: 0000003ff7fcffff
CapBnd: 0000003ff7fcffff
CapAmb: 0000000000000000

app process
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003ff7fcffff
CapAmb: 0000000000000000

anthraxx linux hardened + namespcase + isolation

main process
CapInh: 0000000000000000
CapPrm: 0000003ff7fcffff
CapEff: 0000003ff7fcffff
CapBnd: 0000003ff7fcffff
CapAmb: 0000000000000000

app process
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000

non-root + patch + anthraxx linux hardened + namespcase

main process
CapInh: 00000000000004c0
CapPrm: 00000000000004c0
CapEff: 00000000000004c0
CapBnd: 0000003ff7fcffff
CapAmb: 00000000000004c0

app process
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003ff7fcffff
CapAmb: 0000000000000000

non-root + patch + anthraxx linux hardened + namespcase + isolation

main process
CapInh: 00000000000004c0
CapPrm: 00000000000004c0
CapEff: 00000000000004c0
CapBnd: 0000003ff7fcffff
CapAmb: 00000000000004c0

app process
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000

Actual problem unit not started with mode "non-root + patch + anthraxx linux hardened + namespcase + isolation". Need activate kernel.unprivileged_userns_clone

[Izorkin]

Actual problem unit not started with mode "non-root + patch + anthraxx linux hardened + namespcase + isolation". Need activate kernel.unprivileged_userns_clone

If use unit without isolation and kernel.unprivileged_userns_clone = 0 - worked

Unit configuration with isolation

      {
        "listeners": {
          "*:8373": {
            "application": "php_73"
          }
        },
        "applications": {
          "php_73": {
            "type": "php 7.3",
            "processes": 1,
            "user": "nginx",
            "group": "nginx",
            "root": "/var/www",
            "index": "index.php",
            "isolation": {
              "namespaces": {
                "cgroup": true,
                "credential": true,
                "mount": true,
                "pid": true,
                "uname": true
              },
              "uidmap": [
                {"container": 60, "host": 60, "size": 1}
              ],
              "gidmap": [
                {"container": 60, "host": 60, "size": 1}
              ]
            },
            "options": {
              "admin": {
                "max_execution_time": "30",
                "max_input_time": "30",
                "display_errors": "on",
                "display_startup_errors": "on",
              }
            }
          }
        }
      }

without isolation

      {
        "listeners": {
          "*:8373": {
            "application": "php_73"
          }
        },
        "applications": {
          "php_73": {
            "type": "php 7.3",
            "processes": 1,
            "user": "nginx",
            "group": "nginx",
            "root": "/var/www",
            "index": "index.php",
            "options": {
              "admin": {
                "max_execution_time": "30",
                "max_input_time": "30",
                "display_errors": "on",
                "display_startup_errors": "on",
              }
            }
          }
        }
      }

[i4ki]

Hi @Izorkin

The kernel.unprivileged_userns_clone option controls whether processes can create new user namespaces without privileges (without root in practice).

If you run unitd as root, the flag kernel.unprivileged_userns_clone doesn't do anything.
If you run unitd as non-root and you want to create namespaces, you need the flag set to kernel.unprivileged_userns_clone = 1.

Be aware that this flag is only for the credential namespace and if this isolation is present, all other namespace flags can be used without privileges as well. If you try to use pid namespace alone, for example, and Unit is unprivileged, it's not gonna work.

[Izorkin]

This must resolve issue? - anthraxx/linux-hardened#36

[i4ki]

Hi @Izorkin,

Yes, having a separate capability to enable CLONE_NEWUSER for specific processes would be much better.