Bug: failure to renew port forwarding shuts down the container

[dmitry-t7ko]

Is this urgent?

Yes

Host OS

endeavouros

CPU arch

x86_64

VPN service provider

ProtonVPN

What are you using to run the container

docker-compose

What is the version of Gluetun

Running version latest built on 2024-05-09T14:36:40.530Z (commit ce642a6)

What's the problem 🤔

Marked urgent, because container shuts itself down.

Occasionally with proton vpn port forwarding setup (on either wireguard or openvpn), I see that container shut down after the following error:

ERROR port forwarding loop crashed: stopping previous service: blocking previous port in firewall: removing allowed port 51876 on interface tun0: command failed: "iptables --delete INPUT -i tun0 -p tcp --dport 51876 -j ACCEPT": iptables: Bad rule (does a matching rule exist in that chain?).: exit status 1

This issue is intermittent, and usually appears within 5 minutes of starting container. All the other services connected via gluetun are rendered inaccessible.

My understanding is that port forwarding is being renewed on proton server every 60 seconds, and it that fails gluetun double-frees the iptables rule for the port being forwarded.

Note: I've been seeing other failures to connect to proton, which usually look like this:

gluetun  | 2024-05-09T22:26:51Z INFO [dns] downloading DNS over TLS cryptographic files
gluetun  | 2024-05-09T22:27:06Z WARN [dns] cannot update files: Get "https://www.internic.net/domain/named.root": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
gluetun  | 2024-05-09T22:27:06Z INFO [dns] attempting restart in 10s

and repeating. It might be related to me pulling a lot of data in short amount of time, and proton throttling me (even though running wireguard config on host - without port forwarding - seems to be ok). I'm not sure what to do with it yet, but I can try and provide more info if needed.

Share your logs (at least 10 lines)

gluetun  | ========================================
gluetun  | ========================================
gluetun  | =============== gluetun ================
gluetun  | ========================================
gluetun  | =========== Made with ❤ by ============
gluetun  | ======= https://github.com/qdm12 =======
gluetun  | ========================================
gluetun  | ========================================
gluetun  | 
gluetun  | Running version latest built on 2024-05-09T14:36:40.530Z (commit ce642a6)
gluetun  | 
gluetun  | 🔧 Need help? https://github.com/qdm12/gluetun/discussions/new
gluetun  | 🐛 Bug? https://github.com/qdm12/gluetun/issues/new
gluetun  | ✨ New feature? https://github.com/qdm12/gluetun/issues/new
gluetun  | ☕ Discussion? https://github.com/qdm12/gluetun/discussions/new
gluetun  | 💻 Email? quentin.mcgaw@gmail.com
gluetun  | 💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
gluetun  | 2024-05-09T21:48:09Z INFO [routing] default route found: interface eth0, gateway 192.168.55.1, assigned IP 192.168.55.10 and family v4
gluetun  | 2024-05-09T21:48:09Z INFO [routing] local ethernet link found: eth0
gluetun  | 2024-05-09T21:48:09Z INFO [routing] local ipnet found: 192.168.55.0/24
gluetun  | 2024-05-09T21:48:09Z INFO [firewall] enabling...
gluetun  | 2024-05-09T21:48:09Z INFO [firewall] enabled successfully
gluetun  | 2024-05-09T21:48:10Z INFO [storage] merging by most recent 19425 hardcoded servers and 19425 servers read from /gluetun/servers.json
gluetun  | 2024-05-09T21:48:10Z INFO Alpine version: 3.19.1
gluetun  | 2024-05-09T21:48:10Z INFO OpenVPN 2.5 version: 2.5.8
gluetun  | 2024-05-09T21:48:10Z INFO OpenVPN 2.6 version: 2.6.8
gluetun  | 2024-05-09T21:48:10Z INFO Unbound version: 1.20.0
gluetun  | 2024-05-09T21:48:10Z INFO IPtables version: v1.8.10
gluetun  | 2024-05-09T21:48:10Z INFO Settings summary:
gluetun  | ├── VPN settings:
gluetun  | |   ├── VPN provider settings:
gluetun  | |   |   ├── Name: custom
gluetun  | |   |   ├── Server selection settings:
gluetun  | |   |   |   ├── VPN type: wireguard
gluetun  | |   |   |   ├── Target IP address: [REDACTED]
gluetun  | |   |   |   └── Wireguard selection settings:
gluetun  | |   |   |       ├── Endpoint IP address: [REDACTED]
gluetun  | |   |   |       ├── Endpoint port: 51820
gluetun  | |   |   |       └── Server public key: [REDACTED]
gluetun  | |   |   └── Automatic port forwarding settings:
gluetun  | |   |       ├── Redirection listening port: disabled
gluetun  | |   |       ├── Use code for provider: protonvpn
gluetun  | |   |       └── Forwarded port file path: /tmp/gluetun/forwarded_port
gluetun  | |   └── Wireguard settings:
gluetun  | |       ├── Private key: wAe...3o=
gluetun  | |       ├── Interface addresses:
gluetun  | |       |   └── 10.2.0.2/32
gluetun  | |       ├── Allowed IPs:
gluetun  | |       |   ├── 0.0.0.0/0
gluetun  | |       |   └── ::/0
gluetun  | |       └── Network interface: tun0
gluetun  | |           └── MTU: 1400
gluetun  | ├── DNS settings:
gluetun  | |   ├── Keep existing nameserver(s): no
gluetun  | |   ├── DNS server address to use: 127.0.0.1
gluetun  | |   └── DNS over TLS settings:
gluetun  | |       ├── Enabled: yes
gluetun  | |       ├── Update period: every 24h0m0s
gluetun  | |       ├── Unbound settings:
gluetun  | |       |   ├── Authoritative servers:
gluetun  | |       |   |   └── cloudflare
gluetun  | |       |   ├── Caching: yes
gluetun  | |       |   ├── IPv6: no
gluetun  | |       |   ├── Verbosity level: 1
gluetun  | |       |   ├── Verbosity details level: 0
gluetun  | |       |   ├── Validation log level: 0
gluetun  | |       |   ├── System user: root
gluetun  | |       |   └── Allowed networks:
gluetun  | |       |       ├── 0.0.0.0/0
gluetun  | |       |       └── ::/0
gluetun  | |       └── DNS filtering settings:
gluetun  | |           ├── Block malicious: yes
gluetun  | |           ├── Block ads: no
gluetun  | |           ├── Block surveillance: no
gluetun  | |           └── Blocked IP networks:
gluetun  | |               ├── 127.0.0.1/8
gluetun  | |               ├── 10.0.0.0/8
gluetun  | |               ├── 172.16.0.0/12
gluetun  | |               ├── 192.168.0.0/16
gluetun  | |               ├── 169.254.0.0/16
gluetun  | |               ├── ::1/128
gluetun  | |               ├── fc00::/7
gluetun  | |               ├── fe80::/10
gluetun  | |               ├── ::ffff:127.0.0.1/104
gluetun  | |               ├── ::ffff:10.0.0.0/104
gluetun  | |               ├── ::ffff:169.254.0.0/112
gluetun  | |               ├── ::ffff:172.16.0.0/108
gluetun  | |               └── ::ffff:192.168.0.0/112
gluetun  | ├── Firewall settings:
gluetun  | |   └── Enabled: yes
gluetun  | ├── Log settings:
gluetun  | |   └── Log level: info
gluetun  | ├── Health settings:
gluetun  | |   ├── Server listening address: 127.0.0.1:9999
gluetun  | |   ├── Target address: cloudflare.com:443
gluetun  | |   ├── Duration to wait after success: 2m0s
gluetun  | |   ├── Read header timeout: 100ms
gluetun  | |   ├── Read timeout: 500ms
gluetun  | |   └── VPN wait durations:
gluetun  | |       ├── Initial duration: 1m0s
gluetun  | |       └── Additional duration: 30s
gluetun  | ├── Shadowsocks server settings:
gluetun  | |   └── Enabled: no
gluetun  | ├── HTTP proxy settings:
gluetun  | |   └── Enabled: no
gluetun  | ├── Control server settings:
gluetun  | |   ├── Listening address: :8000
gluetun  | |   └── Logging: yes
gluetun  | ├── OS Alpine settings:
gluetun  | |   ├── Process UID: 1000
gluetun  | |   └── Process GID: 1000
gluetun  | ├── Public IP settings:
gluetun  | |   ├── Fetching: every 12h0m0s
gluetun  | |   ├── IP file path: /tmp/gluetun/ip
gluetun  | |   └── Public IP data API: ipinfo
gluetun  | └── Version settings:
gluetun  |     └── Enabled: yes
gluetun  | 2024-05-09T21:48:10Z INFO [routing] default route found: interface eth0, gateway 192.168.55.1, assigned IP 192.168.55.10 and family v4
gluetun  | 2024-05-09T21:48:10Z INFO [routing] adding route for 0.0.0.0/0
gluetun  | 2024-05-09T21:48:10Z INFO [firewall] setting allowed subnets...
gluetun  | 2024-05-09T21:48:10Z INFO [routing] default route found: interface eth0, gateway 192.168.55.1, assigned IP 192.168.55.10 and family v4
gluetun  | 2024-05-09T21:48:10Z INFO [dns] using plaintext DNS at address 1.1.1.1
gluetun  | 2024-05-09T21:48:10Z INFO [http server] http server listening on [::]:8000
gluetun  | 2024-05-09T21:48:10Z INFO [firewall] allowing VPN connection...
gluetun  | 2024-05-09T21:48:10Z INFO [healthcheck] listening on 127.0.0.1:9999
gluetun  | 2024-05-09T21:48:10Z INFO [wireguard] Using available kernelspace implementation
gluetun  | 2024-05-09T21:48:10Z INFO [wireguard] Connecting to 149.88.20.129:51820
gluetun  | 2024-05-09T21:48:10Z INFO [wireguard] Wireguard setup is complete. Note Wireguard is a silent protocol and it may or may not work, without giving any error message. Typically i/o timeout errors indicate the Wireguard connection is not working.
gluetun  | 2024-05-09T21:48:10Z INFO [dns] downloading DNS over TLS cryptographic files
gluetun  | 2024-05-09T21:48:11Z INFO [healthcheck] healthy!
gluetun  | 2024-05-09T21:48:11Z INFO [dns] downloading hostnames and IP block lists
gluetun  | 2024-05-09T21:48:14Z INFO [dns] init module 0: validator
gluetun  | 2024-05-09T21:48:14Z INFO [dns] init module 1: iterator
gluetun  | 2024-05-09T21:48:14Z INFO [dns] start of service (unbound 1.20.0).
gluetun  | 2024-05-09T21:48:15Z INFO [dns] generate keytag query _ta-4a5c-4f66. NULL IN
gluetun  | 2024-05-09T21:48:15Z INFO [dns] generate keytag query _ta-4a5c-4f66. NULL IN
gluetun  | 2024-05-09T21:48:15Z INFO [dns] ready
gluetun  | 2024-05-09T21:48:15Z INFO [ip getter] Public IP address is [REDACTED]
gluetun  | 2024-05-09T21:48:15Z INFO [vpn] You are running on the bleeding edge of latest!
gluetun  | 2024-05-09T21:48:15Z INFO [port forwarding] starting
gluetun  | 2024-05-09T21:48:15Z INFO [port forwarding] gateway external IPv4 address is [REDACTED]
gluetun  | 2024-05-09T21:48:16Z INFO [port forwarding] port forwarded is 51876
gluetun  | 2024-05-09T21:48:16Z INFO [firewall] setting allowed input port 51876 through interface tun0...
gluetun  | 2024-05-09T21:48:16Z INFO [port forwarding] writing port file /tmp/gluetun/forwarded_port
gluetun  | 2024-05-09T21:50:41Z INFO [healthcheck] healthy!
gluetun  | 2024-05-09T21:55:12Z INFO [firewall] removing allowed port 51876...
gluetun  | 2024-05-09T21:55:12Z ERROR [port forwarding] adding port mapping: executing remote procedure call: connection timeout: failed attempts: read udp 10.2.0.2:44773->10.2.0.1:5351: i/o timeout (tries 1, 2, 3, 4, 5, 6, 7, 8, 9)
gluetun  | 2024-05-09T21:55:12Z INFO [port forwarding] stopping
gluetun  | 2024-05-09T21:55:12Z INFO [firewall] removing allowed port 51876...
gluetun  | 2024-05-09T21:55:12Z ERROR port forwarding loop crashed: stopping previous service: blocking previous port in firewall: removing allowed port 51876 on interface tun0: command failed: "iptables --delete INPUT -i tun0 -p tcp --dport 51876 -j ACCEPT": iptables: Bad rule (does a matching rule exist in that chain?).: exit status 1
gluetun  | 2024-05-09T21:55:12Z INFO dns ticker: terminated ✔
gluetun  | 2024-05-09T21:55:12Z INFO updater ticker: terminated ✔
gluetun  | 2024-05-09T21:55:12Z INFO http server: terminated ✔
gluetun  | 2024-05-09T21:55:12Z INFO control: terminated ✔
gluetun  | 2024-05-09T21:55:12Z INFO updater: terminated ✔
gluetun  | 2024-05-09T21:55:12Z INFO tickers: terminated ✔
gluetun  | 2024-05-09T21:55:12Z INFO HTTP health server: terminated ✔
gluetun  | 2024-05-09T21:55:13Z WARN vpn: goroutine shutdown timed out: after 1s ⚠
gluetun  | 2024-05-09T21:55:13Z INFO shadowsocks proxy: terminated ✔
gluetun  | 2024-05-09T21:55:13Z INFO http proxy: terminated ✔
gluetun  | 2024-05-09T21:55:13Z INFO unbound: terminated ✔
gluetun  | 2024-05-09T21:55:13Z INFO other: terminated ✔
gluetun  | 2024-05-09T21:55:13Z INFO [routing] routing cleanup...
gluetun  | 2024-05-09T21:55:13Z INFO [routing] default route found: interface eth0, gateway 192.168.55.1, assigned IP 192.168.55.10 and family v4
gluetun  | 2024-05-09T21:55:13Z INFO [routing] deleting route for 0.0.0.0/0
gluetun  | 2024-05-09T21:55:13Z ERROR ordered shutdown timed out: vpn: goroutine shutdown timed out: after 1s
gluetun  | 2024-05-09T21:55:13Z INFO Shutdown successful

Share your configuration

services:
  gluetun:
    image: qmcgaw/gluetun
    container_name: gluetun
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun:/dev/net/tun
    ports:
      - 8100:8100 
      - 8110:8080 
      - "3333:3333"
      - "3334:3334/tcp"
      - "3334:3334/udp"
      # - 8888:8888/tcp # HTTP proxy
      # - 8388:8388/tcp # Shadowsocks
      # - 8388:8388/udp # Shadowsocks
    environment:
      - VPN_TYPE=wireguard
      - VPN_SERVICE_PROVIDER=custom
      - VPN_ENDPOINT_IP=[REDACTED]
      - VPN_ENDPOINT_PORT=[REDACTED]
      - WIREGUARD_PUBLIC_KEY=[REDACTED]
      - WIREGUARD_PRIVATE_KEY=[REDACTED]
      - WIREGUARD_ADDRESSES=[REDACTED]
      - VPN_PORT_FORWARDING=on 
      - VPN_PORT_FORWARDING_PROVIDER=protonvpn

[github-actions]

@qdm12 is more or less the only maintainer of this project and works on it in his free time.
Please:

• do not ask for updates, be patient
• 👍 the issue to show your support instead of commenting
  @qdm12 usually checks issues at least once a week, if this is a new urgent bug,
  revert to an older tagged container image

[qdm12]

I also saw that yesterday actually, it's due to nf_tables misbehaving (I reported the bug to the netfilter project) which is now the default backend for iptables since the upgrade to Alpine 3.19. I have a local fix which prefers using the legacy version of iptables (not using nf_tables). I'll push it later today, in the meantime use :v3.38

[qdm12]

Actually the fix was pushed yesterday in commit ce642a6 so just re-pull the latest image and it should be fixed. I'll close this assuming this is resolved.

[github-actions]

Closed issues are NOT monitored, so commenting here is likely to be not seen.
If you think this is still unresolved and have more information to bring, please create another issue.

This is an automated comment setup because @qdm12 is the sole maintainer of this project
which became too popular to monitor issues closed.