Fix `nf_conntrack` table overflow causing UDP packet drops

[josecelano]

Relates to: #26 (comment)

Problem Description

The tracker is experiencing low uptime on newtrackon.com. It could be due to UDP packet drops caused by the Linux netfilter connection tracking (nf_conntrack) table becoming full. This is evidenced by kernel messages showing:

nf_conntrack: nf_conntrack: table full, dropping packet

This issue was discovered while investigating the tracker uptime problem reported in issue #26, specifically detailed in this comment.

Evidence from System Logs

The following kernel messages show the nf_conntrack table overflow:

sudo journalctl -k | grep -i 'udp\|drop\|icmp'
[sudo] password for torrust: 
Jun 17 17:39:30 torrust-demo kernel: DMI: DigitalOcean Droplet/Droplet, BIOS 20171212 12/12/2017
Jun 17 17:39:30 torrust-demo kernel: UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
Jun 17 17:39:30 torrust-demo kernel: UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
Jun 17 17:39:30 torrust-demo kernel: drop_monitor: Initializing network drop monitor service
Jun 17 18:44:13 torrust-demo kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Jun 17 18:49:13 torrust-demo kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Jun 17 18:50:11 torrust-demo kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Jun 17 19:27:38 torrust-demo kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Jun 17 19:34:48 torrust-demo kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Jun 17 19:46:51 torrust-demo kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Jun 17 19:51:47 torrust-demo kernel: nf_conntrack: nf_conntrack: table full, dropping packet

Connection tracking statistics show high early_drop counts and search_restart values, indicating table pressure:

sudo conntrack -S
cpu=0   	found=0 invalid=156 insert=0 insert_failed=0 drop=0 early_drop=814 error=0 search_restart=5374122 (null)=0 (null)=0 
cpu=1   	found=0 invalid=168 insert=0 insert_failed=0 drop=0 early_drop=681 error=0 search_restart=5388730 (null)=0 (null)=0 
cpu=2   	found=0 invalid=190 insert=0 insert_failed=0 drop=0 early_drop=1058 error=0 search_restart=5092846 (null)=0 (null)=0 
cpu=3   	found=562 invalid=27152 insert=0 insert_failed=135 drop=237 early_drop=26755924 error=5 search_restart=503791899 (null)=2539351 (null)=0

Impact

• UDP tracker requests are being dropped at the kernel level
• This causes timeouts from monitoring services like newtrackon.com
• Results in poor uptime statistics (currently around 90% instead of target 95%+)
• Prevents the tracker from being included in the newtrackon.com public tracker list

Proposed Solution

Configure the Linux system to handle the high UDP connection load by adjusting netfilter connection tracking parameters:

1. Increase nf_conntrack table size

# Check current values
cat /proc/sys/net/netfilter/nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_buckets

# Increase the maximum number of connections (e.g., to 262144)
echo 'net.netfilter.nf_conntrack_max = 262144' >> /etc/sysctl.conf

# Increase hash table buckets (should be max/4)
echo 'net.netfilter.nf_conntrack_buckets = 65536' >> /etc/sysctl.conf

2. Reduce connection tracking timeout for UDP

# Reduce UDP timeout from default 30s to 10s for faster cleanup
echo 'net.netfilter.nf_conntrack_udp_timeout = 10' >> /etc/sysctl.conf
echo 'net.netfilter.nf_conntrack_udp_timeout_stream = 10' >> /etc/sysctl.conf

3. Alternative: Disable connection tracking for tracker port

If connection tracking is not needed for the tracker service, we could disable it specifically for the tracker port:

# Add iptables rules to disable connection tracking for UDP port 6969
iptables -t raw -A PREROUTING -p udp --dport 6969 -j NOTRACK
iptables -t raw -A OUTPUT -p udp --sport 6969 -j NOTRACK

4. Apply changes

# Apply sysctl changes
sysctl -p

# Make iptables rules persistent (if using option 3)
iptables-save > /etc/iptables/rules.v4

Monitoring

After implementing the fix, monitor:

• Kernel logs for nf_conntrack messages: sudo journalctl -k | grep nf_conntrack
• Connection tracking statistics: sudo conntrack -S
• Tracker uptime on newtrackon.com

References

• Related issue: Tracker uptime in newtrackon.com is too low (60.80%) #26 (Tracker uptime in newtrackon.com is too low)
• Discovery comment: Tracker uptime in newtrackon.com is too low (60.80%) #26 (comment)
• Linux netfilter documentation: https://www.netfilter.org/documentation/HOWTO/netfilter-hacking-HOWTO.html

Priority

High - This directly affects the tracker's availability and prevents inclusion in public tracker lists.

cc @da2ce7

[Dahrkael]

Connections to/from the tracker don't need to be tracked assuming the firewall is correctly configured to allow traffic in and out on the ports used. Option 3 is the way to go in this case:

IPv4 UDP (port 6969)

iptables -t raw -A PREROUTING -p udp --dport 6969 -j NOTRACK
iptables -t raw -A OUTPUT -p udp --sport 6969 -j NOTRACK

IPv6 UDP (port 6969)

ip6tables -t raw -A PREROUTING -p udp --dport 6969 -j NOTRACK
ip6tables -t raw -A OUTPUT -p udp --sport 6969 -j NOTRACK

IPv4 HTTP (port 80)

iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK
iptables -t raw -A OUTPUT -p tcp --sport 80 -j NOTRACK

IPv6 HTTP (port 80)

ip6tables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK
ip6tables -t raw -A OUTPUT -p tcp --sport 80 -j NOTRACK

IPv4 HTTPS (port 443)

iptables -t raw -A PREROUTING -p tcp --dport 443 -j NOTRACK
iptables -t raw -A OUTPUT -p tcp --sport 443 -j NOTRACK

IPv6 HTTPS (port 443)

ip6tables -t raw -A PREROUTING -p tcp --dport 443 -j NOTRACK
ip6tables -t raw -A OUTPUT -p tcp --sport 443 -j NOTRACK

[josecelano]

I'm going to disable connection tracking.

Before disabling it:

sudo conntrack -S
cpu=0   	found=0 invalid=267 insert=0 insert_failed=0 drop=0 early_drop=2062 error=0 search_restart=9838945 (null)=0 (null)=0 
cpu=1   	found=0 invalid=322 insert=0 insert_failed=0 drop=0 early_drop=1725 error=0 search_restart=9868600 (null)=0 (null)=0 
cpu=2   	found=0 invalid=312 insert=0 insert_failed=0 drop=0 early_drop=2377 error=0 search_restart=9298974 (null)=0 (null)=0 
cpu=3   	found=1322 invalid=28675 insert=0 insert_failed=160 drop=352 early_drop=57052761 error=8 search_restart=907587506 (null)=4561477 (null)=0 

sudo journalctl -k | grep -i 'udp\|drop\|icmp'
[sudo] password for torrust: 
Jun 17 17:39:30 torrust-demo kernel: DMI: DigitalOcean Droplet/Droplet, BIOS 20171212 12/12/2017
Jun 17 17:39:30 torrust-demo kernel: UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
Jun 17 17:39:30 torrust-demo kernel: UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
Jun 17 17:39:30 torrust-demo kernel: drop_monitor: Initializing network drop monitor service
Jun 17 18:44:13 torrust-demo kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Jun 17 18:49:13 torrust-demo kernel: nf_conntrack: nf_conntrack: table full, dropping packet
...

[josecelano]

I'm getting an error trying to disable tracking:

iptables -t raw -A PREROUTING -p udp --dport 6868 -j NOTRACK
iptables -t raw -A OUTPUT -p udp --sport 6868 -j NOTRACK
iptables v1.8.7 (nf_tables): unknown option "--dport"
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.8.7 (nf_tables): unknown option "--sport"
Try `iptables -h' or 'iptables --help' for more information.

It seems I'm using iptables-nft, a more modern version that supports IPv4 + IPv6 in one shot.

[josecelano]

I'm disabling connection tracking only for UDP. I'm not sure if disabling tracking for HTTP ports could affect the firewall. Although I'm using the Digital Ocean firewall and it should not matter at all.

[josecelano]

I have installed iptables-persistentwith:

sudo apt install iptables-persistent

However, finally, I'm not using it because I'm using nftables

[josecelano]

I'm trying to set up the rules in /etc/nftables.conf with nftables:

sudo apt install nftables
# Edit the file
sudo vim /etc/nftables.conf
# Apply and reload rules
sudo systemctl restart nftables
# Verify rules are active
sudo nft list ruleset
# Verify is not being tracked
sudo conntrack -F
sudo conntrack -L | grep 6969
# Reboot the server
sudo reboot
# Confirm nftables rules are active
sudo nft list ruleset | grep notrack

Previous /etc/nftables.conf version:

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
        chain input {
                type filter hook input priority 0;
        }
        chain forward {
                type filter hook forward priority 0;
        }
        chain output {
                type filter hook output priority 0;
        }
}

New /etc/nftables.conf version:

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0;
    }
    chain forward {
        type filter hook forward priority 0;
    }
    chain output {
        type filter hook output priority 0;
    }
}

# Disable connection tracking for UDP ports 6868 and 6969
table inet raw {
    chain prerouting {
        type filter hook prerouting priority raw;
        udp dport {6868, 6969} notrack
    }
    chain output {
        type filter hook output priority raw;
        udp sport {6868, 6969} notrack
    }
}

However, after rebooting the server I don't see the rules:

# Empty output
sudo nft list ruleset | grep notrack

And the server is still tracking:

udp      17 4 src=x.168.68.13 dst=y.19.0.5 sport=52942 dport=6969 [UNREPLIED] src=y.19.0.5 dst=x.168.68.13 sport=6969 dport=52942 mark=0 use=1
...

[josecelano]

I have executed these 3 commands:

sudo systemctl enable nftables         # Enable service at boot
sudo systemctl start nftables          # Start it immediately
sudo systemctl status nftables         # Confirm it's now active

sudo systemctl enable nftables
Created symlink /etc/systemd/system/sysinit.target.wants/nftables.service → /lib/systemd/system/nftables.service.

sudo systemctl start nftables 

sudo systemctl status nftables
● nftables.service - nftables
     Loaded: loaded (/lib/systemd/system/nftables.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2025-06-23 10:10:02 UTC; 4s ago
       Docs: man:nft(8)
             http://wiki.nftables.org
    Process: 6339 ExecStart=/usr/sbin/nft -f /etc/nftables.conf (code=exited, status=0/SUCCESS)
   Main PID: 6339 (code=exited, status=0/SUCCESS)
        CPU: 29ms

Jun 23 10:10:02 torrust-demo systemd[1]: Starting nftables...
Jun 23 10:10:02 torrust-demo systemd[1]: Finished nftables.

sudo nft list ruleset | grep notrack

sudo nft list ruleset | grep notrack
		udp dport { 6868, 6969 } notrack
		udp sport { 6868, 6969 } notrack

[josecelano]

I have rebooted the server:

sudo conntrack -L | grep udp | grep 6969

And the output is empty. It appears that we are no longer tracking UDP port 6969.

[josecelano]

Make sure only nftables is working and iptables is disabled:

sudo iptables -V
sudo iptables -L

iptables v1.8.7 (nf_tables)
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (7 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.20.0.2           tcp dpt:7070
ACCEPT     udp  --  anywhere             172.20.0.2           udp dpt:6969
ACCEPT     udp  --  anywhere             172.20.0.2           udp dpt:6868
ACCEPT     tcp  --  anywhere             172.20.0.2           tcp dpt:1212
ACCEPT     tcp  --  anywhere             172.20.0.3           tcp dpt:9090
ACCEPT     tcp  --  anywhere             172.20.0.4           tcp dpt:3001
ACCEPT     tcp  --  anywhere             172.20.0.5           tcp dpt:3000
ACCEPT     tcp  --  anywhere             172.19.0.2           tcp dpt:3000
ACCEPT     tcp  --  anywhere             172.20.0.6           tcp dpt:https
ACCEPT     tcp  --  anywhere             172.20.0.6           tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (4 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

[josecelano]

The UDP tracker is not working after the changes:

TORRUST_CHECKER_CONFIG='{
    "udp_trackers": ["144.126.245.19:6969"],
    "http_trackers": ["https://tracker.torrust-demo.com"],
    "health_checks": ["https://tracker.torrust-demo.com/api/health_check"]
}' cargo run -p torrust-tracker-client --bin tracker_checker

    Finished `dev` profile [optimized + debuginfo] target(s) in 0.09s
     Running `target/debug/tracker_checker`
2025-06-23T11:08:04.512696Z  INFO torrust_tracker_client::console::clients::checker::service: Running checks for trackers ...

[
  {
    "Health": {
      "Ok": {
        "url": "https://tracker.torrust-demo.com/api/health_check",
        "result": {
          "Ok": "200 OK"
        }
      }
    }
  },
  {
    "Http": {
      "Ok": {
        "url": "https://tracker.torrust-demo.com/",
        "results": [
          [
            "Announce",
            {
              "Ok": null
            }
          ],
          [
            "Scrape",
            {
              "Ok": null
            }
          ]
        ]
      }
    }
  },
  {
    "Udp": {
      "Err": {
        "remote_addr": "144.126.245.19:6969",
        "results": [
          [
            "Setup",
            {
              "Ok": null
            }
          ],
          [
            "Connect",
            {
              "Err": "Failed to receive a connect response, with error: Timeout while waiting for the socket to become readable."
            }
          ]
        ]
      }
    }
  }
]

[josecelano]

The notrack rules don't work with the current Docker configuration. I had to disable them:

sudo vim /etc/nftables.conf # comment the notrack rules. See below.
sudo nft flush ruleset
sudo nft -f /etc/nftables.conf
sudo systemctl restart docker

# Disable connection tracking for UDP ports 6868 and 6969
#table inet raw {
#    chain prerouting {
#        type filter hook prerouting priority raw;
#        udp dport {6868, 6969} notrack
#    }
#    chain output {
#        type filter hook output priority raw;
#        udp sport {6868, 6969} notrack
#    }
#}

[josecelano]

It seems it could work with network mode "host". I will implement this and retry to add the notrack rules.