Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DietPi-Software | WireGuard: wg-quick[....]: RTNETLINK answers: File exists #2491

Closed
curiosity-seeker opened this issue Feb 3, 2019 · 63 comments
Labels
Enhancement 💨 External bug 🐞 For bugs which are not caused by DietPi. Information ℹ️ Solution available 🥂 Definite solution has been done
Milestone

Comments

@curiosity-seeker
Copy link

curiosity-seeker commented Feb 3, 2019

ADMIN EDIT

Solution

  • If you use OpenVPN or PiVPN as well, first try to edit /etc/wireguard/wg0.conf and replace all 10.8.0.X with 10.9.0.X as by default OpenVPN (tun0 interface) uses the same address range. If systemctl restart wg-quick@wg0 succeeds then, do the same with the client configs: /etc/wireguard/wg0-client.conf
  • If it still fails with the same RTNETLINK answers: File exists error, do the following:
sed -i '/^ListenPort[[:blank:]]/a\Table = off' /etc/wireguard/wg0.conf
systemctl restart wg-quick@wg0

For the devs: Enhancement

  • Use different subnets for OpenVPN and WireGuard by default, to avoid interferences, e.g. 10.9.0.0 for WireGuard?
  • Check the same for the ports, although default should be a standard VPN port, in case other random ports are blocked by the network (work place, university and such). Not urgent, since we ask for desired port during install. Perhaps a hint, to use a different one than OpenVPN uses, if not even this is too obvious 😉.

Creating a bug report/issue

Required Information

  • DietPi version:
G_DIETPI_VERSION_CORE=6
G_DIETPI_VERSION_SUB=20
G_DIETPI_VERSION_RC=6
G_GITBRANCH=master
G_GITOWNER=Fourdee

Steps to reproduce

  1. I updated DietPi successfully to v. 6.20.6 (or actually 6.20.7)
  2. I selected Wireguard in dietpi-software, confirmed the proposed configuration for a Wireguard server and rebooted.

Expected behaviour

Wireguard should be running.

Actual behaviour

systemctl status wg-quick@wg0 says:

root@DietPi:~# systemctl status wg-quick@wg0
● wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0
   Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2019-02-03 15:29:04 CET; 10min ago
     Docs: man:wg-quick(8)
           man:wg(8)
           https://www.wireguard.com/
           https://www.wireguard.com/quickstart/
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg-quick.8
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg.8
  Process: 2363 ExecStart=/usr/bin/wg-quick up wg0 (code=exited, status=2)
 Main PID: 2363 (code=exited, status=2)

Feb 03 15:29:04 DietPi wg-quick[2363]: [#] wg setconf wg0 /dev/fd/63
Feb 03 15:29:04 DietPi wg-quick[2363]: [#] ip address add 10.8.0.1/24 dev wg0
Feb 03 15:29:04 DietPi wg-quick[2363]: [#] ip link set mtu 1420 up dev wg0
Feb 03 15:29:04 DietPi wg-quick[2363]: [#] ip route add 10.8.0.0/24 dev wg0
Feb 03 15:29:04 DietPi wg-quick[2363]: RTNETLINK answers: File exists
Feb 03 15:29:04 DietPi wg-quick[2363]: [#] ip link delete dev wg0
Feb 03 15:29:04 DietPi systemd[1]: wg-quick@wg0.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 03 15:29:04 DietPi systemd[1]: Failed to start WireGuard via wg-quick(8) for wg0.
Feb 03 15:29:04 DietPi systemd[1]: wg-quick@wg0.service: Unit entered failed state.
Feb 03 15:29:04 DietPi systemd[1]: wg-quick@wg0.service: Failed with result 'exit-code'.

Extra details

/etc/wireguard/wg0 looks like this:

[Interface]
Address = 10.8.0.1/24
PrivateKey = xyz***
ListenPort = 51820

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o $(sed -n 3p /DietPi/dietpi/.network) -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o $(sed -n 3p /DietPi/dietpi/.network) -j MASQUERADE

[Peer]
PublicKey = abc***
AllowedIPs = 10.8.0.0/24

/lib/systemd/system/wg-quick@.service looks like this:

Unit]
Description=WireGuard via wg-quick(8) for %I
After=network-online.target
Wants=network-online.target
Documentation=man:wg-quick(8)
Documentation=man:wg(8)
Documentation=https://www.wireguard.com/
Documentation=https://www.wireguard.com/quickstart/
Documentation=https://git.zx2c4.com/WireGuard/about/src/tools/man/wg-quick.8
Documentation=https://git.zx2c4.com/WireGuard/about/src/tools/man/wg.8

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/wg-quick up %i
ExecStop=/usr/bin/wg-quick down %i

[Install]
WantedBy=multi-user.target
@MichaIng
Copy link
Owner

MichaIng commented Feb 4, 2019

@curiosity-seeker
Thanks for your report.

The config looks as desired. To sort it out, can you check/verify the wireguard kernel module being active: lsmod

I stumble over: wg-quick[2363]: RTNETLINK answers: File exists
Found: https://raspberrypi.stackexchange.com/questions/13895/solving-rtnetlink-answers-file-exists-when-running-ifup

Try:

systemctl stop wg-quick@wg0
ip addr flush dev wg0
systemctl start wg-quick@wg0

Perhaps also wg-quick up wg0 (running the start command directly from console) shows some more error output.

But also: status=2/INVALIDARGUMENT
Not sure if wg0 is the expected invalid argument in later steps, if the interface was not bought up by the script in the first place due to above error, e.g. in PostUp = iptables -A FORWARD -i %i ...
Further debugging could be done by commenting the PostUp/PostDown lines.


Out of topic:
@Fourdee
Might this be actually also the reason for WiFi + Ethernet interface failure when both are activated? Seeing the above link, both must not have (the same) gateway. Matches our/your thoughts on this, that a routing table is required. Anyway a special config is required when two network devices are about to be used parallel, but if one is active accidentally, perhaps removing one gateway line completely from the /etc/network/interfaces file template allows the interfaces to start at least? Not sure if the gateway directive is simply active for all interfaces, so it can be added once to the end of the file for both?

@curiosity-seeker
Copy link
Author

curiosity-seeker commented Feb 4, 2019

@MichaIng : I executed as you suggested

systemctl stop wg-quick@wg0
ip addr flush dev wg0

and got the message:
Device "wg0" does not exist.

And executing
wg-quick up wg0
got:

[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip address add 10.8.0.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] ip route add 10.8.0.0/24 dev wg0
RTNETLINK answers: File exists
[#] ip link delete dev wg0

@MichaIng
Copy link
Owner

MichaIng commented Feb 4, 2019

@curiosity-seeker

Device "wg0" does not exist.
Ah jep makes sense, since wg-quick adds/removes this interface completely. At least this verifies that indeed the interface is removed and not active/added by any other script, before running next wg-quick.

Did you try to comment the PostUp/PostDown lines in /etc/wireguard/wg0?

I will also do some test install later this night.

@MichaIng MichaIng changed the title Wireguard does not start DietPi-Software | WireGuard: wg-quick[....]: RTNETLINK answers: File exists Feb 4, 2019
@curiosity-seeker
Copy link
Author

Did you try to comment the PostUp/PostDown lines in /etc/wireguard/wg0?

Yes, I did! But I got the same messages as in my post above.

@FabioEight
Copy link

Same issue here! Same setup, same errors:

feb 06 19:32:16 DietPi wg-quick[1835]: [#] wg setconf wg0 /dev/fd/63 feb 06 19:32:16 DietPi wg-quick[1835]: [#] ip address add 10.8.0.1/24 dev wg0 feb 06 19:32:16 DietPi wg-quick[1835]: [#] ip link set mtu 1420 up dev wg0 feb 06 19:32:16 DietPi wg-quick[1835]: [#] ip route add 10.8.0.0/24 dev wg0 feb 06 19:32:16 DietPi wg-quick[1835]: RTNETLINK answers: File exists feb 06 19:32:16 DietPi wg-quick[1835]: [#] ip link delete dev wg0 feb 06 19:32:16 DietPi systemd[1]: wg-quick@wg0.service: Main process exited, code=exited, status=2/INVALIDARGUMENT feb 06 19:32:17 DietPi systemd[1]: Failed to start WireGuard via wg-quick(8) for wg0.
Latest release of DietPi on RPI3B+.

@MichaIng
Copy link
Owner

MichaIng commented Feb 6, 2019

@curiosity-seeker @XRay437
Strange, on my test VM, it does not attempt to add a route:

root@VM-Stretch:~# journalctl -t wg-quick
-- Logs begin at Wed 2019-02-06 20:53:13 CET, end at Wed 2019-02-06 21:45:05 CET. --
Feb 06 21:34:55 VM-Stretch wg-quick[5469]: [#] ip link add wg0 type wireguard
Feb 06 21:34:55 VM-Stretch wg-quick[5469]: [#] wg setconf wg0 /dev/fd/63
Feb 06 21:34:55 VM-Stretch wg-quick[5469]: [#] ip address add 10.8.0.1/24 dev wg0
Feb 06 21:34:56 VM-Stretch wg-quick[5469]: [#] ip link set mtu 1420 up dev wg0
Feb 06 21:34:56 VM-Stretch wg-quick[5469]: [#] iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o $(sed -n 3p /DietPi/dietpi/.network) -j MASQUERADE

In your cases additionally it attempts and fails to: [#] ip route add 10.8.0.0/24 dev wg0 🤔

Related /usr/bin/wg-quick code lines:

cmd_up() {
        local i
        [[ -z $(ip link show dev "$INTERFACE" 2>/dev/null) ]] || die "\`$INTERFACE' already exists"
        trap 'del_if; exit' INT TERM EXIT
        execute_hooks "${PRE_UP[@]}"
        add_if
        set_config
        for i in "${ADDRESSES[@]}"; do
                add_addr "$i"
        done
        set_mtu_up
        set_dns
        for i in $(while read -r _ i; do for i in $i; do [[ $i =~ ^[0-9a-z:.]+/[0-9]+$ ]] && echo "$i"; done; done < <(wg show "$INTERFACE" allowed-ips) | sort -nr -k 2 -t /); do
                add_route "$i"
        done
        execute_hooks "${POST_UP[@]}"
        trap - INT TERM EXIT
}
...
add_route() {
        [[ $TABLE != off ]] || return 0

        if [[ -n $TABLE && $TABLE != auto ]]; then
                cmd ip route add "$1" dev "$INTERFACE" table "$TABLE"
        elif [[ $1 == */0 ]]; then
                add_default "$1"
        else
                [[ $(ip route get "$1" 2>/dev/null) == *dev\ $INTERFACE\ * ]] || cmd ip route add "$1" dev "$INTERFACE"
        fi
}
...
                if [[ $interface_section -eq 1 ]]; then
                        case "$key" in
                        Address) ADDRESSES+=( ${value//,/ } ); continue ;;
                        MTU) MTU="$value"; continue ;;
                        DNS) DNS+=( ${value//,/ } ); continue ;;
                        Table) TABLE="$value"; continue ;;
                        PreUp) PRE_UP+=( "$value" ); continue ;;
                        PreDown) PRE_DOWN+=( "$value" ); continue ;;
                        PostUp) POST_UP+=( "$value" ); continue ;;
                        PostDown) POST_DOWN+=( "$value" ); continue ;;
                        SaveConfig) read_bool SAVE_CONFIG "$value"; continue ;;
                        esac
                fi

If I could follow everything correctly, the following is responsible:
[[ $(ip route get "$1" 2>/dev/null) == *dev\ $INTERFACE\ * ]] || cmd ip route add "$1" dev "$INTERFACE"

Check on my system with wg0 up (service started):

root@VM-Stretch:~# ip route get 10.8.0.0/24
broadcast 10.8.0.0 dev wg0 src 10.8.0.1
    cache <local,brd>

With wg0 down (service stopped):

root@VM-Stretch:~# ip route get 10.8.0.0/24
10.8.0.0 via 192.168.178.1 dev eth0 src 192.168.178.29
    cache
  • Strange, this does not match *dev\ $INTERFACE\ *, which should be dev wg0 , so the route should be added as well 🤔... Perhaps this changes with the earlier ip commands in the row...

Manually applying interface steps:

root@VM-Stretch:~# ip link add wg0 type wireguard
root@VM-Stretch:~# ip route get 10.8.0.0/24
10.8.0.0 via 192.168.178.1 dev eth0 src 192.168.178.29
    cache
root@VM-Stretch:~# wg setconf wg0 /dev/fd/63
fopen: No such file or directory
root@VM-Stretch:~# ip address add 10.8.0.1/24 dev wg0
root@VM-Stretch:~# ip route get 10.8.0.0/24
10.8.0.0 via 192.168.178.1 dev eth0 src 192.168.178.29
    cache
root@VM-Stretch:~# ip link set mtu 1420 up dev wg0
root@VM-Stretch:~# ip route get 10.8.0.0/24
broadcast 10.8.0.0 dev wg0 src 10.8.0.1
    cache <local,brd>
  • Interesting, after applying MTU, the route matches *dev\ $INTERFACE\ *, thus ip route add 10.8.0.0/24 dev wg0 is skipped.
  • If I try:
root@VM-Stretch:~# ip route add 10.8.0.0/24 dev wg0
RTNETLINK answers: File exists

So it seems, that somehow the ip route get check fails or gives wrong result, so wg-quick tries to add it as separate step, which fails, since the route indeed does already exist.

  • Perhaps on some network adapters, it takes a little while, until the route is established, so the check returns false first, but when ip route add is called, it is there?

@MichaIng
Copy link
Owner

MichaIng commented Feb 6, 2019

@curiosity-seeker @XRay437
Based on the above research, please try and paste output of the following, calling the commands one by one (to allow some time in between):

ip link del wg0 # Remove possibly existing interface
ip link add wg0 type wireguard
ip address add 10.8.0.1/24 dev wg0
ip link set mtu 1420 up dev wg0
ip route get 10.8.0.0/24

@FabioEight
Copy link

@curiosity-seeker @XRay437
Based on the above research, please try and paste output of the following, calling the commands one by one (to allow some time in between):

ip link del wg0 # Remove possibly existing interface
ip link add wg0 type wireguard
ip address add 10.8.0.1/24 dev wg0
ip link set mtu 1420 up dev wg0
ip route get 10.8.0.0/24

Hello, here we are:

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
 ────────────────────────────────────────────────
 DietPi | Thu 07/02/19 - 10:25
 ────────────────────────────────────────────────
 v6.20.6 | RPi 3 Model B+ (armv7l)
 ────────────────────────────────────────────────
 eth0 | 192.168.X.Y
 ────────────────────────────────────────────────
 DietPi Team     : Daniel Knight (founder), MichaIng, K-Plan
 Image           : DietPi Core Team (pre-image: Raspbian Lite)
 Web             : https://DietPi.com | https://twitter.com/dietpi_
 Patreon Legends : PINE64 community
 Donate          : https://DietPi.com/#donate
 DietPi Hosting  : Powered by https://MyVirtualServer.com

 dietpi-launcher  = All the DietPi programs in one place.
 dietpi-config    = Feature rich configuration tool for your device.
 dietpi-software  = Select optimized software for installation.
 htop             = Resource monitor.
 cpu              = Shows CPU information and stats.

root@DietPi:~# ip link del wg0
Cannot find device "wg0"
root@DietPi:~# ip link add wg0 type wireguard
root@DietPi:~# ip address add 10.9.0.1/24 dev wg0
root@DietPi:~# ip link set mtu 1420 up dev wg0
root@DietPi:~# ip route get 10.9.0.0/24
broadcast 10.9.0.0 dev wg0 src 10.9.0.1 uid 0 
    cache <local,brd> 
root@DietPi:~# systemctl start wg-quick@wg0
Job for wg-quick@wg0.service failed because the control process exited with error code.
See "systemctl status wg-quick@wg0.service" and "journalctl -xe" for details.
root@DietPi:~# systemctl status wg-quick@wg0
● wg-quick@wg0.service - WireGuard via wg-qui
ck(8) for wg0
   Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since T
hu 2019-02-07 10:27:47 GMT; 11s ago
     Docs: man:wg-quick(8)
           man:wg(8)
           https://www.wireguard.com/
           https://www.wireguard.com/quickstart/
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg-quick.8
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg.8
  Process: 14918 ExecStart=/usr/bin/wg-quick up wg0 
����������������������������������������������������(code=exited, status=1/FAILURE)
 Main PID: 14918 (code=exited, status=1/FAILURE)

feb 07 10:27:46 DietPi systemd[1]: Starting WireGuard via wg-quick(8) for wg0...
feb 07 10:27:47 DietPi wg-quick[14918]: wg-quick: `wg0' already exists
feb 07 10:27:47 DietPi systemd[1]: wg-quick@wg0.
service: Main process exited, code=exited, status=1/FAILURE
feb 07 10:27:47 DietPi systemd[1]: Failed to sta
rt WireGuard via wg-quick(8) for wg0.
feb 07 10:27:47 DietPi systemd[1]: wg-quick@wg0.
service: Unit entered failed state.
feb 07 10:27:47 DietPi systemd[1]: wg-quick@wg0.
service: Failed with result 'exit-code'.

Please, notice that I'm using the subnet 10.9.0.0/24 instead of 10.8.0.0/24 because on that subnet I've already set the tun0 if of PiVPN (in use with pi-Hole).
Thanks!

@MichaIng
Copy link
Owner

MichaIng commented Feb 7, 2019

@XRay437

ip route get 10.9.0.0/24
broadcast 10.9.0.0 dev wg0 src 10.9.0.1 uid 0
cache <local,brd>

So indeed the required route is set up as desired and the service should not try to add it again. Indeed I guess it simply needs to wait a tiny bit longer.
This is an external bug from my point of view that we should forward to the WireGuard devs.

Note that the above steps are for debugging, no solution that would enable you to start the service. The problem it you completely need to remove the link to be able to start the service, so no chance to do any pre-configuration as solution, AFAIK.

But we could simply add a sleep 1 to the binary itself. Quite nice that everything is pure shell 😃.
So try the following:

ip link del wg0 # Clean up from the testing before
sed -i '/[[:blank:]]set_mtu_up/a\\tsleep 1' /usr/bin/wg-quick
grep 'sleep' /usr/bin/wg-quick # Verify it's there
systemctl start wg-quick@wg0.service

@MichaIng MichaIng added the External bug 🐞 For bugs which are not caused by DietPi. label Feb 7, 2019
@curiosity-seeker
Copy link
Author

@MichaIng : I can report the same result as @XRay437 .

Unfortunately your latest suggestion didn't fix the problem:

root@DietPi:~# ip link del wg0
root@DietPi:~# sed -i '/[[:blank:]]set_mtu_up/a\\tsleep 1' /usr/bin/wg-quick
root@DietPi:~# grep 'sleep' /usr/bin/wg-quick
        sleep 1
root@DietPi:~# systemctl start wg-quick@wg0.service
Job for wg-quick@wg0.service failed because the control process exited with error code.
See "systemctl status wg-quick@wg0.service" and "journalctl -xe" for details.
root@DietPi:~# systemctl status wg-quick@wg0.service
● wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0
   Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2019-02-07 12:15:12 CET; 16s ago
     Docs: man:wg-quick(8)
           man:wg(8)
           https://www.wireguard.com/
           https://www.wireguard.com/quickstart/
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg-quick.8
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg.8
  Process: 3498 ExecStart=/usr/bin/wg-quick up wg0 (code=exited, status=2)
 Main PID: 3498 (code=exited, status=2)

Feb 07 12:15:10 DietPi wg-quick[3498]: [#] wg setconf wg0 /dev/fd/63
Feb 07 12:15:11 DietPi wg-quick[3498]: [#] ip address add 10.8.0.1/24 dev wg0
Feb 07 12:15:11 DietPi wg-quick[3498]: [#] ip link set mtu 1420 up dev wg0
Feb 07 12:15:12 DietPi wg-quick[3498]: [#] ip route add 10.8.0.0/24 dev wg0
Feb 07 12:15:12 DietPi wg-quick[3498]: RTNETLINK answers: File exists
Feb 07 12:15:12 DietPi wg-quick[3498]: [#] ip link delete dev wg0
Feb 07 12:15:12 DietPi systemd[1]: wg-quick@wg0.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 07 12:15:12 DietPi systemd[1]: Failed to start WireGuard via wg-quick(8) for wg0.
Feb 07 12:15:12 DietPi systemd[1]: wg-quick@wg0.service: Unit entered failed state.
Feb 07 12:15:12 DietPi systemd[1]: wg-quick@wg0.service: Failed with result 'exit-code'.

And repeating the commands suggested further above got:

root@DietPi:~# systemctl stop wg-quick@wg0
root@DietPi:~# ip addr flush dev wg0
Device "wg0" does not exist.
root@DietPi:~# wg-quick up wg0
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip address add 10.8.0.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] ip route add 10.8.0.0/24 dev wg0
RTNETLINK answers: File exists
[#] ip link delete dev wg0

@MichaIng
Copy link
Owner

MichaIng commented Feb 7, 2019

@curiosity-seeker
Okay thanks for testing.

Checking again the code:

add_route() {
        [[ $TABLE != off ]] || return 0

        if [[ -n $TABLE && $TABLE != auto ]]; then
                cmd ip route add "$1" dev "$INTERFACE" table "$TABLE"
        elif [[ $1 == */0 ]]; then
                add_default "$1"
        else
                [[ $(ip route get "$1" 2>/dev/null) == *dev\ $INTERFACE\ * ]] || cmd ip route add "$1" dev "$INTERFACE"
        fi
}
  • $TABLE is definitely not set, this requires an additional command argument or special config file entry.
  • The net mask is /24 not /0
  • When setting the MTU manually, the route definitely exists, so (re)adding should be skipped.
  • 🤔... Waiting (sleeping) longer than 1 second should be very unlikely required.
  • Why not setting TABLE='off' to forcefully skip adding the route? Checking which other effects this has and where exactly to add...

@MichaIng
Copy link
Owner

MichaIng commented Feb 7, 2019

@curiosity-seeker @XRay437
Okay please try the following:

sed -i '/^[[:blank:]]*sleep 1/d' /usr/bin/wg-quick # Revert adding the sleep, if already added.
sed -i '/^ListenPort[[:blank:]]/a\Table = off' /etc/wireguard/wg0.conf # Actively disable the routing table to force skipping "ip route add"
systemctl restart wg-quick@wg0

Btw good hint about the double use of 10.8.0.0 for WireGuard and OpenVPN. Usually you only want to use one VPN, but for testing and such both might be wanted as well. Perhaps we should by default switch WireGuard to 10.9.0.0 then?

@curiosity-seeker
Copy link
Author

Success !!!!

root@DietPi:~# ip link del wg0
Cannot find device "wg0"
root@DietPi:~# sed -i '/^[[:blank:]]*sleep 1/d' /usr/bin/wg-quick
root@DietPi:~# sed -i '/^ListenPort[[:blank:]]/a\Table = off' /etc/wireguard/wg0.conf
root@DietPi:~# systemctl restart wg-quick@wg0
root@DietPi:~# systemctl status wg-quick@wg0.service
● wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0
   Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2019-02-07 12:45:12 CET; 10s ago
     Docs: man:wg-quick(8)
           man:wg(8)
           https://www.wireguard.com/
           https://www.wireguard.com/quickstart/
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg-quick.8
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg.8
  Process: 3837 ExecStart=/usr/bin/wg-quick up wg0 (code=exited, status=0/SUCCESS)
 Main PID: 3837 (code=exited, status=0/SUCCESS)

Feb 07 12:45:12 DietPi systemd[1]: Starting WireGuard via wg-quick(8) for wg0...
Feb 07 12:45:12 DietPi wg-quick[3837]: [#] ip link add wg0 type wireguard
Feb 07 12:45:12 DietPi wg-quick[3837]: [#] wg setconf wg0 /dev/fd/63
Feb 07 12:45:12 DietPi wg-quick[3837]: [#] ip address add 10.8.0.1/24 dev wg0
Feb 07 12:45:12 DietPi wg-quick[3837]: [#] ip link set mtu 1420 up dev wg0
Feb 07 12:45:12 DietPi wg-quick[3837]: [#] iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o $(sed -n 3p /DietPi/dietpi/.network) -j MASQUERADE
Feb 07 12:45:12 DietPi systemd[1]: Started WireGuard via wg-quick(8) for wg0.

Great!!!

@MichaIng
Copy link
Owner

MichaIng commented Feb 7, 2019

@curiosity-seeker
Many thanks for reporting back. Please test if client connections work as desired, so we are sure we can for now promote this as solution, if users run into this, e.g. adding as hint to the dietpi.com docs.

And why ever the route check goes wrong, we should forward this to WireGuard devs.

@curiosity-seeker
Copy link
Author

Btw good hint about the double use of 10.8.0.0 for WireGuard and OpenVPN. Usually you only want to use one VPN, but for testing and such both might be wanted as well. Perhaps we should by default switch WireGuard to 10.9.0.0 then?

So I should modify this in /etc/wireguard/wg0.conf and /etc/wireguard/wg0-client.conf as I've been using OpenVPN, too, shouldn't I?

@curiosity-seeker
Copy link
Author

Please test if client connections work as desired, so we are sure we can for now promote this as solution, if users run into this, e.g. adding as hint to the dietpi.com docs.

I will certainly try to do this ASAP but it might take some time as I'm busy with other things right now.

@MichaIng
Copy link
Owner

MichaIng commented Feb 7, 2019

@curiosity-seeker
Hmm in case of an conflict, you should have shown other error messages before, AFAIK.
Use: ip a s tun0 to check your OpenVPN address.

@curiosity-seeker
Copy link
Author

root@DietPi:~# ip a s tun0
3: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none 
    inet 10.8.0.1/24 brd 10.8.0.255 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::4d14:b08a:89d6:8267/64 scope link flags 800 
       valid_lft forever preferred_lft forever

And wireguard is using 10.8.0.2 so there shouldn't be a conflict,

@MichaIng
Copy link
Owner

MichaIng commented Feb 7, 2019

@curiosity-seeker
Hmm, not sure but generally I would use a different subnet for both, otherwise OpenVPN connections might be routed into the WireGuard net or the other way round. I am no expert in networking, but this might be even the reason for the error in the first place, if OpenVPN already added the 10.8.0.0/XX route.

Since XRay437 faced the same issue with already different subnets, it might not be the only thing, but when you find time, just to sort it out, you could try the following:

  • nano /etc/wireguard/wg0.conf
  • Remove or comment the Table = off line
  • Instead change all 10.8.0.X to 10.9.0.X. This would be required for the clients as well of course.
  • Retry systemctl restart wg-quick@wg0

If it fails with the same error, readd/uncomment Table = off. The different address ranges should stay in place definitely to avoid interferences.

I will certainly try to do this ASAP but it might take some time as I'm busy with other things right now.

Jep no worries, you guys already helped a lot with testing and reporting back 😃.

@FabioEight
Copy link

FabioEight commented Feb 7, 2019

@curiosity-seeker @XRay437
Okay please try the following:

sed -i '/^[[:blank:]]*sleep 1/d' /usr/bin/wg-quick # Revert adding the sleep, if already added.
sed -i '/^ListenPort[[:blank:]]/a\Table = off' /etc/wireguard/wg0.conf # Actively disable the routing table to force skipping "ip route add"
systemctl restart wg-quick@wg0

Btw good hint about the double use of 10.8.0.0 for WireGuard and OpenVPN. Usually you only want to use one VPN, but for testing and such both might be wanted as well. Perhaps we should by default switch WireGuard to 10.9.0.0 then?

root@DietPi:~# ip link del wg0
root@DietPi:~# sed -i '/^[[:blank:]]*sleep 1/d' /usr/bin/wg-quick
root@DietPi:~# sed -i '/^ListenPort[[:blank:]]/a\Table = off' /etc/wireguard/wg0.conf
root@DietPi:~# systemctl restart wg-quick@wg0
Job for wg-quick@wg0.service failed because the control process exited with error code.
See "systemctl status wg-quick@wg0.service" and "journalctl -xe" for details.
root@DietPi:~# systemctl status wg-quick@wg0
● wg-quick@wg0.service - WireGuard via wg-qui
ck(8) for wg0
   Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since T
hu 2019-02-07 12:05:40 GMT; 15s ago
     Docs: man:wg-quick(8)
           man:wg(8)
           https://www.wireguard.com/
           https://www.wireguard.com/quickstart/
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg-quick.8
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg.8
  Process: 15576 ExecStart=/usr/bin/wg-quick up wg0 
����������������������������������������������������(code=exited, status=1/FAILURE)
 Main PID: 15576 (code=exited, status=1/FAILURE)

feb 07 12:05:39 DietPi wg-quick[15576]: [#] wg setconf wg0 /dev/fd/63
feb 07 12:05:39 DietPi wg-quick[15576]: [#] ip address add 10.9.0.1/24 dev wg0
feb 07 12:05:39 DietPi wg-quick[15576]: [#] ip link set mtu 1420 up dev wg0
feb 07 12:05:39 DietPi wg-quick[15576]: [#] iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o $(sed -n 3p /DietPi/dietpi/.network) -j MASQUERADE
feb 07 12:05:39 DietPi wg-quick[15576]: iptables: Invalid argument. Run `dmesg' for more information.
feb 07 12:05:39 DietPi wg-quick[15576]: [#] ip link delete dev wg0
feb 07 12:05:40 DietPi systemd[1]: wg-quick@wg0.
service: Main process exited, code=exited, status=1/FAILURE
feb 07 12:05:40 DietPi systemd[1]: Failed to sta
rt WireGuard via wg-quick(8) for wg0.
feb 07 12:05:40 DietPi systemd[1]: wg-quick@wg0.
service: Unit entered failed state.
feb 07 12:05:40 DietPi systemd[1]: wg-quick@wg0.
service: Failed with result 'exit-code'.
```
Now I see an error related to iptables, maybe something is wrong on my wg0.conf? I changed every 10.8.X.X in 10.9.X.X (also for wg0-client) so I think it should be fine!

@curiosity-seeker
Copy link
Author

@MichaIng : I said I'm busy with other things but I simply couldn't wait ;-)

I added the public key and 10.8.0.2/24 to the Wireguard app on my iphone - and everything seems to work as it should!

I first tried it using my Wifi, and then disabled Wifi on the iPhone - all is well!

@MichaIng
Copy link
Owner

MichaIng commented Feb 7, 2019

@XRay437
Okay just to verify the route is up, then check which iptables command fails:

ip route get 10.9.0.0/24
iptables -A FORWARD -i wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o $(sed -n 3p /DietPi/dietpi/.network) -j MASQUERADE

And as suggested within the error output, check the last lines of: dmesg


@curiosity-seeker
Great it works. However I would still suggest to switch to 10.9.0.X and test as mentioned above, if you find time: https://github.com/Fourdee/DietPi/issues/2491#issuecomment-461393237

@FabioEight
Copy link

FabioEight commented Feb 7, 2019

@XRay437
Okay just to verify the route is up, then check which iptables command fails:

ip route get 10.9.0.0/24
iptables -A FORWARD -i wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o $(sed -n 3p /DietPi/dietpi/.network) -j MASQUERADE

And as suggested within the error output, check the last lines of: dmesg

@curiosity-seeker
Great it works. However I would still suggest to switch to 10.9.0.X and test as mentioned above, if you find time: #2491 (comment)

Ok, I'm using wg0.conf with "#Tables = OFF".

ip route get 10.9.0.0/24:

10.9.0.0 via 192.168.X.X (gateway IP) dev eth0 src 192.168.X.Z (DietPi IP) uid 0 cache

[    7.194126] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    7.200935] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    7.298961] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[    7.298969] brcmfmac: power management disabled
[    7.535333] FS-Cache: Netfs 'cifs' registered for caching
[    7.535558] Key type cifs.spnego registered
[    7.535566] Key type cifs.idmap registered
[    7.643622] wireguard: loading out-of-tree module taints kernel.
[    7.645704] wireguard: WireGuard 0.0.20190123 loaded. See www.wireguard.com for information.
[    7.645712] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[    7.715344] tun: Universal TUN/TAP device driver, 1.6[   10.316975] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   13.831616] CIFS VFS: Error connecting to socket. Aborting operation.
[   13.831629] CIFS VFS: cifs_mount failed w/return code = -113
[   20.071705] CIFS VFS: Error connecting to socket. Aborting operation.
[   20.071721] CIFS VFS: cifs_mount failed w/return code = -113
[ 3370.392676] systemd[1]: certbot.timer: Adding 6h 14min 37.437636s random time.
[ 3370.728813] systemd[1]: certbot.timer: Adding 5h 18min 44.420104s random time.
[ 3370.997193] systemd[1]: certbot.timer: Adding 5h 50min 58.780251s random time.
root@DietPi:~# 

@MichaIng
Copy link
Owner

@curiosity-seeker
Great it works now. Indeed veeeery strange that the WireGuard client was able to connect even with disabled service server-side 🤔. First I though that it might have connected to OpenVPN, but that should be impossible, at least due to wrong keys. Perhaps somehow the interface was still up from the testings before.

Generally to check which interfaces are active: ip a

Okay so I think we can close this issue?

  • Default IPs for WireGuard are switched to 10.9.0.X with v6.22, which was at least in one case the reason for failsing service start, NB: This is not applied on update/reinstall, we do never touch existing WireGuard configs!
  • In cases wrong client DNS entry fixed as well.
  • iptables issue resolved as well with fresh install. apt install --reinstall iptables or apt purge iptables && apt install iptables should work as well, whatever was the reason.
  • We will keep in mind Table = off, although seems to be finally no required, after resolving all OpenVPN/WireGuard overlap.
  • We learned much about how WireGuard inits the interface 😃.

Fell free to add/use or reopen the issue, if required.

@curiosity-seeker
Copy link
Author

curiosity-seeker commented Feb 10, 2019

I hate to say that what I wrote a couple of hours ago is no longer true. While my iPad still appears as 10.9.0.3 in the Pi-hole query log, the iPhone does not anymore. Although I haven't changed anything. It seems to me that Wireguard is not stable enough right now.

EDIT: I can still ping both devices, even the iPhone under 10.9.0.2.
EDIT: Correction. I cannot ping 10.9.0.2:
--- 10.9.0.2 ping statistics --- 12 packets transmitted, 0 received, 100% packet loss, time 11449ms
although the iPhone and its Wireguard app still say that they are connected to the VPN.

@curiosity-seeker
Copy link
Author

Btw, ip a gives

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:d2:df:67 brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.40/24 brd 192.168.178.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ba27:ebff:fed2:df67/64 scope link 
       valid_lft forever preferred_lft forever
3: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none 
    inet 10.8.0.1/24 brd 10.8.0.255 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::b716:83f0:4f23:297a/64 scope link flags 800 
       valid_lft forever preferred_lft forever
10: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.9.0.1/24 scope global wg0
       valid_lft forever preferred_lft forever

@curiosity-seeker
Copy link
Author

curiosity-seeker commented Feb 10, 2019

What also puzzles me: wg gives

interface: wg0
  public key: AAABBBCCC
  private key: (hidden)
  listening port: 51820

peer: XXXYYYZZZ
  endpoint: NNN.BB.AAA:56021
  allowed ips: 10.9.0.0/24
  latest handshake: 5 minutes, 59 seconds ago
  transfer: 18.11 KiB received, 28.05 KiB sent

Why is the endpoint port different from 51820?

@MichaIng
Copy link
Owner

MichaIng commented Feb 10, 2019

@curiosity-seeker

Why is the endpoint port different from 51820?

If I get this correct, 56021 is the WireGuard port of the client. It is expected that this does not necessarily match the one from the server and is does not need to.

E.g. in my case (damn too a while, QR code not readable and manually mixing 1 and l in terminal, which look identical 🤣):

root@VM-Stretch:~# wg
interface: wg0
  public key: XXX
  private key: (hidden)
  listening port: 51820

peer: YYY
  endpoint: 192.168.178.24:51052
  allowed ips: 10.8.0.0/24
  latest handshake: 17 seconds ago
  transfer: 259.88 KiB received, 712.95 KiB sent

My wg interface looks the same:

3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1
    link/none
    inet 10.8.0.1/24 scope global wg0
       valid_lft forever preferred_lft forever

What I recognized is that the WireGuard module client shows a successful connection, even if it was not successful, at least in my case when I entered a wrong key. That is quite a bad behaviour. It should clearly tell you that authentication failed!


About you having inconsistent stable connection:

  • Really I am still sure that OpenVPN and WireGuard running in parallel can have further unexpected issues. So be sure I would disable OpenVPN, at least during WireGuard tests: systemctl disable openvpn
  • Assure that tun0 interface does not show up anymore: ip l
  • Switching to different IP range was one thing, about iptables redirects and other things might still cause problems. And since finally you would want to use OpenVPN or WireGuard only, I think it's not worth it to track everything down to the roots, at least for me lacking still some network basics 😉.

@MichaIng
Copy link
Owner

MichaIng commented Feb 10, 2019

I just realized that the qrencode to create the QR code of course translates the whole file with all comments, not just the relevant/active parts. So all our added comments are included which make the code quite large and e.g. with my camera fails.

Simple solution to only translate non-comment lines:
grep -v '^#' /etc/wireguard/wg0-client.conf | qrencode -t ansiutf8

@curiosity-seeker
Copy link
Author

curiosity-seeker commented Feb 11, 2019

@MichaIng

Why is the endpoint port different from 51820?

If I get this correct, 56021 is the WireGuard port of the client. It is expected that this does not necessarily match the one from the server and is does not need to.

Thanks - understood!

What I recognized is that the WireGuard module client shows a successful connection, even if it was not successful, at least in my case when I entered a wrong key. That is quite a bad behaviour. It should clearly tell you that authentication failed!

Indeed - it seems that Wireguard still needs some polishing. There is a reason why Linus hasn't added it to the kernel yet ....

* Really I am still sure that OpenVPN and WireGuard running in parallel can have further unexpected issues. So be sure I would disable OpenVPN, at least during WireGuard tests: `systemctl disable openvpn`

This is actually what I had done! However, it hasn't improved the situation.

* Assure that tun0 interface does not show up anymore: `ip l`

This is how it looks now:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether b8:27:eb:d2:df:67 brd ff:ff:ff:ff:ff:ff
15: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none

Thanks again for your help! And let's hope that new Wireguard versions will bring the necessary improvements.

@MichaIng
Copy link
Owner

MichaIng commented Feb 11, 2019

@curiosity-seeker
Jep also the Android app is still marked as pre-release.

We can help to fix issues by giving some feedback 😎. I will report the missing error prompt when authentication fails.

Thanks to your testing with tun0/OpenVPN disabled, sorting out incompatibility with OpenVPN, perhaps we can also have the WireGuard server devs have a look at this. Of course I cannot assure that it is not as well a config issue our side, due to lack of deeper knowledge. I as well simply followed various guides and own testing + compare with our OpenVPN setup, to estimate required configs 😉.


While reading I understand better and better how this routing and iptables actually work:

  • The routes added are actually for outgoing requests only to decide which target IP to send to which network interface/adapter. E.g. ip route add 10.9.0.0/24 dev wg0 simply means that all requests to 10.9.0.X IPs should be send to/handled by the wg0 interface instead of e.g. eth0. In my test case:
    root@VM-Stretch:~# ip r
    default via 192.168.178.1 dev eth0 onlink
    10.8.0.0/24 dev wg0 proto kernel scope link src 10.8.0.1
    192.168.178.0/24 dev eth0 proto kernel scope link src 192.168.178.29
    
    • When IP is 10.8.0.X, send to wg0 (This should be based on AllowedIPs = 10.8.0.0/24)
      • Test if changing wg0.conf is reflected here accordingly.
    • When IP is 192.168.178.X, send to eth0
    • Everything else (default) send to default gateway 192.168.178.1 (my router) on eth0
  • This also makes sense the other way round from the client config side:
    • AllowedIPs = 0.0.0.0/0 leads to 0.0.0.0/0 dev wg0-client proto kernel scope link src 10.8.0.2, so when IP is X.X.X.X (so all IPs), then send to wg0-client interface, thus to the VPN server.
    • The other entries respectively.
  • While this only allows to define which requests to send through the tunnel and which not, this alone only allows communication between the two peers, while non will forward any request and both will answer only, if the request target IP matches their own on the related interface.
  • The real magic now indeed is the sysctl ip_forward setting. This enables the system to forward incoming requests that do not match their own IP to the desired target interface/gateway. Since I could not find any rule that explicitly says "forward incoming wg0 requests to eth0/standard gateway", I think the forwarding is done automatically based on the local system routes, as above. So a default gateway/interface is required to forward all requests/target IPs. In other words:
    • ip_forward is required to use the "Forward" lane (chain). Logic actually, but I always mixed up forwarding and routing 😄.
  • Now to the iptables rules:
    • iptables -A FORWARD -i wg0 -j ACCEPT: Accept incoming requests to the forward chain from wg0 interface. This is actually only required, if via iptables by default forwarding requests are blocked. But makes sense to have this set by default, so we are free to define firewall rules without breaking our VPN forwarding.
    • iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE: Masquerade any outgoing request to eth0, which means that the sender will be replaced by the VPN server IP. So that is basically the NAT behaviour, exactly what your router does: Target hosts will only see the router IP (and answer to it) instead of the actual client IP. The latter would also not make any sense, since the local network IP of course is not valid for any non-local host. The correct redirection of the answer to the actual client is most likely part of this functionality/the nat table. "Answers" and incoming "requests" are anyway two different things, although I do not know about the protocol details about that 😉. So this MASQUERADE should be required for any VPN redirection, since neither local network hosts nor www hosts can access/answer the VPN client via its WireGuard interface IP.

@curiosity-seeker
Copy link
Author

@MichaIng : Just out of curiosity : Does it run well on your systems?

@MichaIng
Copy link
Owner

MichaIng commented Feb 11, 2019

@curiosity-seeker
It passed my testing to connect and use DNS + www access from inside and outside local network. But I don't use the VPN for daily live.

Also I did not test with multiple clients, actually.

It is also possible to add each client as a separate peer to the server config, with own key pair. Should not be required actually, but at least could be worth to test. I have to work now but will give you the required instructions later.

My ToDo:

  • Check for multi peer config in our OpenVPN installer

@curiosity-seeker
Copy link
Author

@MichaIng

It is also possible to add each client as a separate peer to the server config, with own key pair. Should not be required actually, but at least could be worth to test.

Yes, this is what I've done. My wg0.conf looks like this:

[Interface]
Address = 10.9.0.1/24
PrivateKey = XXX
ListenPort = 51820
#Table = off

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o $(sed -n 3p /DietPi/dietpi/.network) -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o $(sed -n 3p /DietPi/dietpi/.network) -j MASQUERADE

[Peer]
# iPhone
PublicKey = YYYY
# iPad
PublicKey = ZZZ
AllowedIPs = 10.9.0.0/24

And my wg0-client.conf is this one:

[Interface]
# The address must be unique for each client, use "10.8.0.3/24" for the second client and so on.
#iPhone
Address = 10.9.0.2/24
PrivateKey = AAA
# iPad
Address = 10.9.0.3/24
PrivateKey = BBB
# Comment the following to preserve the clients default DNS server, or force a desired one.
#DNS = 127.0.0.1
DNS = 10.9.0.1

# Kill switch: Uncomment the following, if the client should stop any network traffic, when disconnected from the VPN server
# NB: This requires "iptables" to be installed, thus will most likely not work on mobile phones.
#PostUp = iptables -I OUTPUT ! -o %i -m mark ! --mark  -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -I OUTPUT ! -o %i -m mark ! --mark  -m addrtype ! --dst-type LOCAL -j REJECT
#PreDown = iptables -D OUTPUT ! -o %i -m mark ! --mark  -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -D OUTPUT ! -o %i -m mark ! --mark  -m addrtype ! --dst-type LOCAL -j REJECT

[Peer]
PublicKey = CCC
# Tunnel all network traffic through the VPN:
#       AllowedIPs = 0.0.0.0/0, ::/0
# Tunnel access to server-side local network only:
#       AllowedIPs = 192.168.178.0/24
# Tunnel access to VPN server only:
#       AllowedIPs = 192.168.178.40/32
AllowedIPs = 0.0.0.0/0, ::/0

Is this correct? I just wonder if AllowedIPs in wg0-client.conf should also be 10.9.0.0/24 ...

Btw.: I still have to read your updated post above in order to (hopefully!) understand all intricacies ;-)

@MichaIng
Copy link
Owner

MichaIng commented Feb 11, 2019

@curiosity-seeker
Now in wg0.conf you have still one peer, only allowing two pubic keys. Although I am not sure if not the second key simply overrides the first, so only the second key will successfully connect/authenticate.

To add two peers, you need two [Peer] entries followed by key and IP each.
If one peer is only used by one single client, it could look like this:

[Peer]
PublicKey = YYY
AllowedIPs = 10.9.0.2/32

[Peer]
PublicKey = ZZZ
AllowedIPs = 10.9.0.3/32

For simplicity you should be able to use the same key pair for both clients then.

About the client side allowed IPs, it depends if you want to forward all network traffic through the VPN or only access to the VPN server or local network.
See the comments in wg0-client.conf which AllowedIPs to use in which use case.

@curiosity-seeker
Copy link
Author

curiosity-seeker commented Feb 11, 2019

@MichaIng : You saved my day, my friend!!! I changed wg0.conf according to your instructions - and now I can see my iPhone as 10.9.0.2 in the Pi-hole query log again. Hurrah!

I really should have thought about adding a new [Peer] section after adding a new device.

So I really hope that this is the final solution!

Thank you very much once more !!!

EDIT: And now finally wg reports:

interface: wg0
  public key: AAA
  private key: (hidden)
  listening port: 51820

peer: BBB
  allowed ips: 10.9.0.2/32

peer: CCC
  allowed ips: 10.9.0.3/32

@MichaIng
Copy link
Owner

@curiosity-seeker
Great! So hmm it did not work reliable before to use a single peer with single key (pair) and AllowedIPs = 10.9.0.0/24 and using this for two clients in parallel?
As said above, I guess adding two PublicKey entries might lead to one overriding the other, thus being the reason why one client with the overridden key fails to authenticate. However I can run tests here as well to verify this.

As long as this now works reliable for you, we might need to change the online docs in how to add multiple clients. Also it looks somehow cleaner if every client has it's own peer 😉.

@curiosity-seeker
Copy link
Author

@MichaIng

So hmm it did not work reliable before to use a single peer with single key (pair) and AllowedIPs = 10.9.0.0/24 and using this for two clients in parallel?

Quite frankly I did not try that. I just had not added a second [Peer] section - that was the problem. Now with 2 key pairs it works well. Yesterday it happened that after some time the iPad didn't appear as 10.9.0.3 anymore (although the VPN symbol was still there and the device was allegedly still connected to the VPN), so I had to deactivate and re-activate it to make it work again. Since then the Wireguard tunnel is stable for both devices.

That problem could be caused by the Wireguard app - but it could also be a general problem with VPN support in iOS: I have been using AdGuard Pro on both devices which creates a local VPN in order to filter ads (before you ask: that local VPN is automatically disabled when switching to OpenVPN or Wireguard). And with that local VPN enabled I sometimes had problems connecting to my Wifi.

@MichaIng
Copy link
Owner

MichaIng commented Feb 12, 2019

@curiosity-seeker
Okay thanks for all your testing and reporting. We take this issue as valuable source for general info and background about WireGuard and VPN setup.

I will also run some tests here and in case change the online docs about how to add multiple clients for best stability: https://github.com/Fourdee/DietPi/issues/2540

@curiosity-seeker
Copy link
Author

curiosity-seeker commented Mar 23, 2019

@MichaIng : One late additional remark FYI. I have been using Wireguard with Pi-hole successfully for the most time. However, every one or two days I noticed that in the Pi-hole log my two iOS devices didn't appear as 10.9.0.2 and 10.9.0.3, resp. - which meant that the Wireguard tunnel was no longer active so I had to re-enable it manually in the iOS app. I enabled Persistent Keepalive and set it to 10 secs - but to no avail.

But then I tripped over this Reddit thread which sounded familiar. I tried the solution presented in the last post - and since then all is well.

I don't know after all those changes above if that is really the default in Dietpi and if you already changed it in 6.22. But I wanted to mention it as it might explain possible problems other users might have.

@MichaIng
Copy link
Owner

MichaIng commented Mar 23, 2019

@curiosity-seeker
Many thanks for this info. Hmm makes sense that forwarding IPv6 requests fails if IPv6 functionality is disabled on the server. But strange that this leads to connection losses instead of the client simply switching to IPv4 in case of failure.

However the question is if this depends on the server only or on the local network as well. Did you disable IPv6 via dietpi-config?
Please paste:

sysctl -a 2> /dev/null | grep 'disable_ipv6'

Hmm, as long is the IPv6 kernel module is enabled, actually forwarding IPv6 should work fine 🤔. Possibly the issue can be solved already by force enabling IPv6 (via sysctl) only for the wg0 interface.

The problem is:

  • If we disable IPv6 forwarding, then accordingly IPv6 requests from the VPN clients will bypass the tunnel. Of course this is not what you want, especially since this is done silently, so you don't even know which connection is tunnelled and which not. Super unsecure and unexpected and nothing we will provide natively!
  • So either we need to assure, that if the server has IPv6 disabled, also the clients will never use IPv6 (which is not possible AFAIK), or we need to force IPv6 being enabled.

Based on how you disabled IPv6, I hope simply setting echo 'net.ipv6.conf.wg0.disable_ipv6 = 0' > ZZ-dietpi-enable_wg0_ipv6.conf will fix the issue.
If the kernel does not support IPv6 (or manually disabled by user), we need to prompt a big warning that clients IPv6 traffic will bypass the VPN.

@curiosity-seeker
Copy link
Author

@MichaIng

root@DietPi:~# sysctl -a 2> /dev/null | grep 'disable_ipv6'
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.wg0.disable_ipv6 = 0

Unfortunately I have to say that I found that a couple of hours ago the Wireguard tunnel for my iPad wasn't active anymore after working flawlessly for several days before. Sigh. So that change isn't a real solution although the situation has improved.

@curiosity-seeker
Copy link
Author

Just noticed that after today's update to 6.22.3 Wiregurad is broken:

root@DietPi:~# systemctl status wg-quick@wg0
Unit wg-quick@wg0.service could not be found.
root@DietPi:~# wg
-bash: wg: command not found.

@MichaIng
Copy link
Owner

MichaIng commented Mar 24, 2019

@curiosity-seeker

Just noticed that after today's update to 6.22.3 Wiregurad is broken:

That is strange. The update does not touch WireGuard installs at all, only fixes the Debian Sid repo implementation: https://github.com/MichaIng/DietPi/blob/master/dietpi/patch_file#L1544-L1557

But there was a WireGuard APT package update, if I remember right.

Could you please paste the update log:

cat /var/tmp/dietpi/logs/dietpi-update.log

About IPv6:

  • Okay so IPv6 it is enabled, kernel- and system-wise for all interfaces 🤔.
  • I just found an earlier paste of your which shows that the interfaces also get an IPv6 address:
root@DietPi:~# ip a s tun0
3: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none 
    inet 10.8.0.1/24 brd 10.8.0.255 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::4d14:b08a:89d6:8267/64 scope link flags 800 
       valid_lft forever preferred_lft forever
  • Just to assure, could you show this for all interfaces (especially wg0 and the main internet interface of course): ip a

@curiosity-seeker
Copy link
Author

@MichaIng
cat /var/tmp/dietpi/logs/dietpi-update.log shows:

[  OK  ] DietPi-Update | wget https://raw.githubusercontent.com/Fourdee/DietPi/master/dietpi/pre-patch_file -O pre-patch_file

 DietPi-Pre-patch
─────────────────────────────────────────────────────
 Mode: Applying critical pre-patches

[ WARN ] Pre-patch 3 | Storing global DietPi-Software password with enhanced security
[  OK  ] Successfully applied critical pre-patches

[  OK  ] DietPi-Update | APT update, please wait...
Get:1 ftp://ftp.fu-berlin.de/linux/raspbian/raspbian stretch InRelease [15.0 kB]
Hit:2 https://archive.raspberrypi.org/debian stretch InRelease
Get:4 ftp://ftp.fu-berlin.de/linux/raspbian/raspbian stretch/main armhf Packages [11.7 MB]
Get:3 https://cdn-aws.deb.debian.org/debian sid InRelease [242 kB]
Get:5 https://cdn-aws.deb.debian.org/debian sid/main armhf Packages [8,054 kB]
Get:6 https://cdn-aws.deb.debian.org/debian sid/main Translation-en [6,312 kB]
Get:7 https://cdn-aws.deb.debian.org/debian sid/main Translation-de [1,663 kB]
Fetched 27.9 MB in 36s (772 kB/s)
Reading package lists...

[  OK  ] DietPi-Update | G_AGUP
[  OK  ] DietPi-Update | APT upgrade, please wait...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following package was automatically installed and is no longer required:
  wireguard-tools
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

[  OK  ] DietPi-Update | G_AGUG
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   119    0   119    0     0    240      0 --:--:-- --:--:-- --:--:--   240
100   120    0   120    0     0    175      0 --:--:-- --:--:-- --:--:--  117k
100 2502k  100 2502k    0     0   960k      0  0:00:02  0:00:02 --:--:-- 2335k
[  OK  ] DietPi-Update | Unpack update archive
rm: cannot remove 'DietPi-master/dietpi/.??*': No such file or directory
[  OK  ] DietPi-Update | Copy DietPi core files to RAMdisk
[  OK  ] DietPi-Update | Copy DietPi rootfs files in place
[  OK  ] DietPi-Update | Set execute permissions for DietPi scripts
[  OK  ] DietPi-Set_software | Connection test: https://raw.githubusercontent.com/MichaIng/DietPi/master/dietpi.txt
[  OK  ] DietPi-Set_software | Patching dietpi.txt
[  OK  ] DietPi-Set_software | Downloading current dietpi.txt
[  OK  ] DietPi-Set_software | dietpi.txt verification completed
[  OK  ] verify_dietpi.txt  | Completed

 DietPi-Update
─────────────────────────────────────────────────────
 Mode: Running incremental patching

[ INFO ] DietPi-Update | Current version : v6.21.1
[ INFO ] DietPi-Update | Latest version  : v6.22.3
[ INFO ] DietPi-Update | Patching 6.21.1 to 6.22
[  OK  ] DietPi-Patchfile | Added setting for i in /etc/bashrc.d/*.sh /etc/bashrc.d/*.bash; do [ -r "$i" ] && . $i; done to end of file /etc/bash.bashrc
[ INFO ] DietPi-Patchfile | APT installation for: p7zip, please wait...
[  OK  ] DietPi-Patchfile | G_AGI p7zip
p7zip-full set to automatically installed.
[ INFO ] DietPi-Patchfile | APT autoremove + purge, please wait...
(Reading database ... 56865 files and directories currently installed.)
Removing p7zip-full (16.02+dfsg-3+deb9u1) ...
Removing wireguard-tools (0.0.20190123-1) ...
dpkg: warning: while removing wireguard-tools, directory '/etc/wireguard' not empty so not removed
(Reading database ... 56703 files and directories currently installed.)
Purging configuration files for p7zip-full (16.02+dfsg-3+deb9u1) ...
[  OK  ] DietPi-Patchfile | G_AGA
HEAD is now at 347994db Merge pull request #903 from pi-hole/release/v4.2
--2019-03-24 12:03:15--  https://dietpi.com/downloads/binaries/rpi/debian-archive-keyring.deb
Resolving dietpi.com (dietpi.com)... 104.27.179.199, 104.27.178.199, 2606:4700:30::681b:b3c7, ...
Connecting to dietpi.com (dietpi.com)|104.27.179.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57392 (56K) [application/x-debian-package]
Saving to: ‘debian-archive-keyring.deb’

     0K .......... .......... .......... .......... .......... 89% 1.52M 0s
    50K ......                                                100% 4.75M=0.03s

2019-03-24 12:03:16 (1.64 MB/s) - ‘debian-archive-keyring.deb’ saved [57392/57392]

(Reading database ... 56703 files and directories currently installed.)
Preparing to unpack debian-archive-keyring.deb ...
Unpacking debian-archive-keyring (2018.1) over (2018.1) ...
Setting up debian-archive-keyring (2018.1) ...
[  OK  ] DietPi-Patchfile | Setting in /DietPi/dietpi.txt adjusted: DEV_GITOWNER=MichaIng
[  OK  ] DietPi-Patchfile | Setting in /DietPi/dietpi/.version adjusted: G_GITOWNER=MichaIng
[  OK  ] DietPi-Software | Initialized database
[  OK  ] DietPi-Software | Reading database
[ SUB2 ] DietPi-Software > Automated reinstall
[ INFO ] DietPi-Software | 108: AmiBerry is not currently installed
[ INFO ] DietPi-Software | The program must be installed, before reinstall can be used
[  OK  ] DietPi-Software | No changes applied for: AmiBerry
[ INFO ] DietPi-Software | 160: Allo_update is not currently installed
[ INFO ] DietPi-Software | The program must be installed, before reinstall can be used
[  OK  ] DietPi-Software | No changes applied for: Allo_update
[  OK  ] DietPi-Update | Patching to v6.22.3 completed

So wireguard-tools were removed, and its reverse dependencies are:

root@DietPi:~# apt-cache rdepends wireguard-tools
wireguard-tools
Reverse Depends:
  wireguard-dkms
  wireguard
  wireguard-dkms

I guess that explains it. dpkg -l | grep wireguard shows that only wireguard-dkms is installed.

And here is the output of ip a:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:d2:df:67 brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.40/24 brd 192.168.178.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ba27:ebff:fed2:df67/64 scope link 
       valid_lft forever preferred_lft forever

So wg0 is missing probably because wireguard was removed.

@MichaIng
Copy link
Owner

@curiosity-seeker
Why ever the wireguard package was removed, at least not during the update. So

G_AGI wireguard
systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0

should resolve it. Your VPN settings should be still in place.

@curiosity-seeker
Copy link
Author

curiosity-seeker commented Mar 24, 2019

@MichaIng
I got the following error first:

The following packages have unmet dependencies:
 wireguard : Depends: wireguard-dkms (= 0.0.20190227-1) but 0.0.20190123-1 is to be installed or
                      wireguard-modules (= 0.0.20190227-1) but it is not installable
E: Unable to correct problems, you have held broken packages.

So I removed wireguard-dkms and installed wireguard anew.

After starting wg-quick@wg0 ip a shows wg0 again.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:d2:df:67 brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.40/24 brd 192.168.178.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ba27:ebff:fed2:df67/64 scope link 
       valid_lft forever preferred_lft forever
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.9.0.1/24 scope global wg0
       valid_lft forever preferred_lft forever

@MichaIng
Copy link
Owner

MichaIng commented Mar 24, 2019

wireguard-dkms (= 0.0.20190227-1) but 0.0.20190123-1 is to be installed

Hmm, possible that priority 99 does not allow auto updates/installs besides when selecting the package or repo manually, e.g. G_AGI wireguard wireguard-dkms wireguard-tools or G_AGI wireguard -t sid 🤔. Will run some tests.

Okay can't test since the old packages are not available anymore. Downloaded the current ones and need to wait for next WireGuard update.

Reading some more about pinning: https://wiki.debian.org/AptPreferences

  • Hard to predict the exact behaviour depending on actually used repo and Raspbian vs Debian. Could be required to pin priority >= 500 to have higher priority than currently installed packages and by this allow auto upgrades.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement 💨 External bug 🐞 For bugs which are not caused by DietPi. Information ℹ️ Solution available 🥂 Definite solution has been done
Projects
None yet
Development

No branches or pull requests

3 participants