Kernel before 5.6.X (April 2020): superuser rights needed / setsockopt: Operation not permitted

[fcmircea]

Hello,

I'm trying to run BindToInterface in a script as part of a systemd service that runs under its own separate user.

The service file looks like this:

[Unit]
Description=Deluge Bittorrent Client Daemon
Documentation=man:deluged
After=network-online.target mnt-storage.mount
Requires=mnt-storage.mount
BindsTo=mnt-storage.mount

[Service]
Type=simple
UMask=000

ExecStart=/bin/bash /media/bti/deluged.sh

Restart=on-failure

# Time to wait before forcefully stopped.
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target

The script is as follows:

#!/bin/bash

BIND_INTERFACE=eno2 DNS_OVERRIDE_IP=8.8.8.8 BIND_EXCLUDE=127.0.0.1,192.168. LD_PRELOAD=/media/bti/bindToInterface.so /usr/bin/deluged -d -l /var/log/deluge/daemon.log -L warning

when executing:

systemctl status deluged

I get the following output:

bash[503711]: setsockopt: Operation not permitted

Is there a way to run your tool without elevated privileges?

Thanks

[JsBergbau]

setsockopt normally works as normal user. However I know that systemd enforces some restrictions, like certain directories are not writeable.

Please try curl --interface eno2 ifconfig.me from normal command prompt and from systemd unit. First try should succeed. Second try should give the same error message. If so, I suggest asking at stackoverflow or superuser regarding setsockopt and systemd. Please report your results.

[fcmircea]

Done, posted at: https://stackoverflow.com/questions/69994803/setsockopt-for-systemd-unit-error-operation-not-permited

Let's see if there are any answers.

[birdie-github]

I've got the same issue:

BIND_INTERFACE=tun0 DNS_OVERRIDE_IP=8.8.8.8 LD_PRELOAD=/usr/local/lib/x86_64-linux-gnu/bindToInterface.so curl https://ifconfig.co
setsockopt: Operation not permitted
setsockopt: Operation not permitted
setsockopt: Operation not permitted
setsockopt: Operation not permitted
curl: (6) Could not resolve host: ifconfig.co

however

curl --interface tun0 https://ifconfig.co

works perfectly. The problem is I have to run a different application which cannot bind to any interfaces and I don't know what to do.

I've found some interesting info and posted it here: https://unix.stackexchange.com/questions/679881/setsockopt-operation-not-permitted-error-when-trying-to-use-bindtointerface-u

[JsBergbau]

Hi birdie-github. Could it be an Apparmor / SELinux issue? You could try disabling this.

Somewhere I've read, that binding to interface needs CAP_NET_RAW privilige. You can try by sudo setcap cap_net_raw <Path of your executable>. For testing with curl this would be

sudo setcap cap_net_raw $(eval readlink -f `which curl`)

To remove all capabilities after testing you can use setcap -r /path/to/program, for more information about removing capabilities see https://unix.stackexchange.com/questions/303423/unset-setcap-additional-capabilities-on-excutable

[birdie-github]

# setcap cap_net_raw+eip `which curl`

# getcap `which curl`
/usr/bin/curl = cap_net_raw+eip

Now it produces no errors but it does NOT work:

curl https://ifconfig.co
144.AAA.BBB.CCC

curl --interface tun0 https://ifconfig.co
45.XXX.YYY.ZZZ

BIND_INTERFACE=tun0 DNS_OVERRIDE_IP=8.8.8.8 LD_PRELOAD=/usr/local/lib/x86_64-linux-gnu/bindToInterface.so curl https://ifconfig.co
144.AAA.BBB.CCC

And it cannot work:

https://stackoverflow.com/questions/18058426/does-using-linux-capabilities-disable-ld-preload

Like Oliver Matthews answered, LD_PRELOAD is disabled for both setuid binaries, and for binaries having file capabilities, for security reasons.

strace however shows that with setcap setsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, "tun0\0", 5) succeeds when using curl without any overrides (i.e. with --interface tun0).

[JsBergbau]

Thanks for giving the hint, that setting capabilities will not work with preloaded libraries.
Can you try

sudo setcap -r /usr/local/lib/x86_64-linux-gnu/bindToInterface.so
sudo chown root /usr/local/lib/x86_64-linux-gnu/bindToInterface.so
sudo chmod u+s /usr/local/lib/x86_64-linux-gnu/bindToInterface.so

and try again. Since the source code is public and it is quite it is ok from a security point of view.

[birdie-github]

# setcap -r `which curl` 
# setcap -r `which strace` 

# chown root /usr/local/lib/x86_64-linux-gnu/bindToInterface.so
# chmod u+s /usr/local/lib/x86_64-linux-gnu/bindToInterface.so
# ls -l /usr/local/lib/x86_64-linux-gnu/bindToInterface.so
-rwsr-xr-x 1 root root 11848 Dec  2 12:32 /usr/local/lib/x86_64-linux-gnu/bindToInterface.so

$ BIND_INTERFACE=tun0 DNS_OVERRIDE_IP=8.8.8.8 LD_PRELOAD=/usr/local/lib/x86_64-linux-gnu/bindToInterface.so curl https://ifconfig.co
setsockopt: Operation not permitted
setsockopt: Operation not permitted
setsockopt: Operation not permitted
setsockopt: Operation not permitted
curl: (6) Could not resolve host: ifconfig.co

(# - running under root, $ - running under a normal user account).

I see no AppArmor related messages in system logs.

[birdie-github]

The following commands:

# setcap cap_net_raw+eip /usr/local/lib/x86_64-linux-gnu/bindToInterface.so
# setcap cap_net_raw+eip `which curl`

make the errors disappear but the issue remains.

[JsBergbau]

I could now reproduce the error and try to find a solution. In the meantime you don't need to try anything.

[JsBergbau]

@birdie-github You seem to have also a lot of linux experience.

Soruce-Code of curl

#ifdef SO_BINDTODEVICE
      /* I am not sure any other OSs than Linux that provide this feature,
       * and at the least I cannot test. --Ben
       *
       * This feature allows one to tightly bind the local socket to a
       * particular interface.  This will force even requests to other
       * local interfaces to go out the external interface.
       *
       *
       * Only bind to the interface when specified as interface, not just
       * as a hostname or ip address.
       *
       * interface might be a VRF, eg: vrf-blue, which means it cannot be
       * converted to an IP address and would fail Curl_if2ip. Simply try
       * to use it straight away.
       */
      if(setsockopt(sockfd, SOL_SOCKET, SO_BINDTODEVICE,
                    dev, (curl_socklen_t)strlen(dev) + 1) == 0) {
        /* This is typically "errno 1, error: Operation not permitted" if
         * you're not running as root or another suitable privileged
         * user.
         * If it succeeds it means the parameter was a valid interface and
         * not an IP address. Return immediately.
         */
        return CURLE_OK;
      }
#endif

So sourcecode of curl expects this to fail without root priviliges. Despite it works.
bindToInterface uses exact the same API-call setsockopt(sockfd, SOL_SOCKET, SO_BINDTODEVICE, &interface, sizeof(interface)); //Will fail if socket is already bound to another interface

Perhaps you have an idea what is different there.

[birdie-github]

Despite it works.

No, it fails, I've checked it using strace:

strace curl --interface tun0 https://ifconfig.co 2>&1 | grep setsockopt
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [60], 4) = 0
setsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, "tun0\0", 5) = -1 EPERM (Operation not permitted)

curl binds to the IP address instead.

Maybe folks at stackexchange have something to say: https://unix.stackexchange.com/questions/679881/setsockopt-operation-not-permitted-error-when-trying-to-use-bindtointerface-u

I've no idea either. Not a lot of actual info on the topic and I'm not a good C programmer to read the kernel source code.

This is definitely an issue with Ubuntu. I cannot reproduce it locally with Fedora 35 despite having SeLinux enabled.

[JsBergbau]

Very strange behaviour
You can test this behaviour even with having a second interface like a VPN by using lo interface. Of course curl can't connect, but the socket binding error will occur before that. That info might help on stackoverflow.

First I thought it depends on the kernel version, because on Raspberry PI with Debian Buster and Kernel version 4.x it fails, whereas it works with kernel version 5.x. However on Ubuntu with Kernel version 5.x it also fails.

Debian Buster amd64 Kernel v4.x

uname -r
4.19.0-9-amd64
strace curl --interface lo ifconfig.me 2>&1 | grep sockopt
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [60], 4) = 0
setsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, "lo\0", 3) = -1 EPERM (Die Operation ist nicht erlaubt)

Debian Buster Kernel 5.x (Raspberry PI)

uname -r
5.10.17-v7+

strace curl --interface lo ifconfig.me 2>&1 | grep sockopt
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [60], 4) = 0
setsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, "lo\0", 3) = 0

Debian Buster Kernel v4.x (Raspberry PI)

 uname -r
4.19.97+

strace curl --interface lo ifconfig.me 2>&1 | grep sockopt
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [60], 4) = 0
setsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, "lo\0", 3) = -1 EPERM (Die Operation ist nicht erlaubt)

Ubuntu 20.04.3 LTS

uname -r
5.4.0-91-generic
strace curl --interface lo ifconfig.me 2>&1 | grep sockopt
setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(5, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
setsockopt(5, SOL_TCP, TCP_KEEPINTVL, [60], 4) = 0
setsockopt(5, SOL_SOCKET, SO_BINDTODEVICE, "lo\0", 3) = -1 EPERM (Operation not permitted)

[birdie-github]

Please update the bug title to indicate that it affects Ubuntu/Debian and not necessarily other Linux distros.

And bash[] in its title is absolutely not necessary.

[birdie-github]

Information (courtesy or https://unix.stackexchange.com/users/325065/uncle-billy ):

It seems that the ability to do that as a regular user is pretty new (5.6.X - April 2020). With older kernels you need CAP_NET_RAW.

Google tells me that the kernel in Ubuntu 18.04.6 is 4.15. So you need to either upgrade or setcap cap_net_raw+ep your_program

Have the library pass the socket fd (via scm_rights on a Unix socket) to a setcap executable which sets that option on it. You cannot gain privileges from a library -- the main executable should be setuid or setcap

So, the issue is down to an old kernel.

I wonder if you could code this feature.

[JsBergbau]

Have the library pass the socket fd (via scm_rights on a Unix socket) to a setcap executable which sets that option on it. You cannot gain privileges from a library -- the main executable should be setuid or setcap

If I understand this correct: You continue executing BIND_INTERFACE=tun0 DNS_OVERRIDE_IP=8.8.8.8 LD_PRELOAD=/usr/local/lib/x86_64-linux-gnu/bindToInterface.so curl https://ifconfig.co
But instead that bindToInterface.so binds that socket, it forwards the request to a service running which then creates the requested socket and passes it in this case to curl. Seems a lot of work for a solution that will eventually not be needed anymore since newer kernels don't need this.

[birdie-github]

Seems a lot of work for a solution that will eventually not be needed anymore since newer kernels don't need this.

Yeah, I'm now thinking of how to install a newer kernel because doing everything else is just too much hassle. This bug report is probably safe to close. Maybe you could add Linux kernel >= 5.6 as a requirement in README.md.

Thank you for your swift response anyways. Really appreciated.

[JsBergbau]

Thank you for your praise.

I've already included that superuser rights are required prior to kernel 5.6.x. Maybe that is also an option for you instead of installing a new kernel that isn't shipped with your Ubuntu version.

[birdie-github]

I need to run chromium, so superuser is not an option (even if the application allowed that, I'd never run a web browser under root - too dangerous). Anyways, installing kernel 5.11 has solved all my issues. Thanks again.

[JsBergbau]

Of course you're right, never run a web browser as root.
If kernel update would be much harder, I'd recommend microsocks https://github.com/rofl0r/microsocks and setting it as browser in chromium. I use this solution with different VPNs (in fact this was the main motivation for developing this code) and it works like a charm.

[fcmircea]

I've tried the 5.11 kernel using the exact setup in the initial post.

Unfortunately, although no error is thrown either via systemctl status or journalctl, the deluge process doesn't seem to be bound to that specific interface.

This I verify by running network activity monitoring tools (eg: speedometer) that clearly show the deluge process throwing traffic across multiple interfaces.

To this end, with kernel 5.11 the deluge process doesn't even seem to obey the metric parameter in network manager configuration.

I had to revert to 5.4 kernel, and unfortunately rely on route metric as an overall system priority for traffic.

Any suggestions greatly appreciated, some exact scenarios to run on 5.11 kernel to verify bindtointerface actually works as part of systemd units.

Thank you!

[JsBergbau]

@fcmircea

What is the result when using

#!/bin/bash

BIND_INTERFACE=eno2 DNS_OVERRIDE_IP=8.8.8.8 BIND_EXCLUDE=127.0.0.1,192.168. LD_PRELOAD=/media/bti/bindToInterface.so /usr/bin/deluged -d -l /var/log/deluge/daemon.log -L warning

without any systemd unit on kernel 5.11?

[fcmircea]

I first stop the daemon via

systemctl stop deluged

deluge runs under its own user so first I had to run

sudo su deluge --shell /bin/bash

I then run

#!/bin/bash

BIND_INTERFACE=enp7s0 DNS_OVERRIDE_IP=8.8.8.8 BIND_EXCLUDE=127.0.0.1,192.168. LD_PRELOAD=/media/bti/bindToInterface.so /usr/bin/deluged -d -l /var/log/deluge/daemon.log -L warning

I get no error message and the daemon starts up and I can connect to it via the web interface

I use nmcli to edit the ipv4.route-metric value to make it equal between the two interfaces

nmcli con edit wc1
set ipv4.route-metric 10
verify
save
activate

route -n shows equal metric between the two iterfaces (enp7s0 and enp0s31f6).

However, both speedometer and nethogs show deluge not bound to the correct interface enp7s0 and instead pushing traffic via enp0s31f6

PID    USER     PROGRAM                 DEV        SENT             RECEIVED
6810  deluge   /usr/bin/python3       enp0s3    1633.005      48.950 KB/sec

speedometer screenshot: https://gyazo.com/15e7a9eaa7d8702393f7d15481feed69

Thank you for looking into this.

PS: Of course:

• on 5.4 kernel I get the setsockopt: Operation not permitted when trying BIND_INTERFACE
• on 5.7 kernel I get the same result as on 5.11

[JsBergbau]

According to https://torguard.net/knowledgebase.php?action=displayarticle&id=215 deluge supports Sock5 proxy. I think the easiest solution is to use microsocks which can be bound to specific interface and setup deluge to use this socks proxy, see https://github.com/JsBergbau/BindToInterface#microsocks