Tracker uptime in newtrackon.com is too low (60.80%)

[josecelano]

Got to https://newtrackon.com/ and search for "torrust" on the right side search box:

We set the right tracker intervals in the tracker configuration (300 secs):

[core.announce_policy]
interval = 300
interval_min = 300

I don't see anything wierd in Grafana or Digital Ocean graphs:

The only reason I see is the latency. Maybe the tracker is too slow sending the response.

If I make a request with the tracker-client I get a fast response:

time cargo run --bin udp_tracker_client announce udp://tracker.torrust-demo.com:6969/announce 9c38422213e30bff212b30c360d26f9a02136422 | jq
    Finished `dev` profile [optimized + debuginfo] target(s) in 0.08s
     Running `/home/josecelano/Documents/git/committer/me/github/torrust/torrust-tracker/target/debug/udp_tracker_client announce 'udp://tracker.torrust-demo.com:6969/announce' 9c38422213e30bff212b30c360d26f9a02136422`
{
  "AnnounceIpv4": {
    "transaction_id": -888840697,
    "announce_interval": 300,
    "leechers": 0,
    "seeders": 1,
    "peers": []
  }
}

real	0m0.333s
user	0m0.101s
sys	0m0.032s

Maybe we could add a new metric: upd_average_response_latency

cc @da2ce7

[josecelano]

I run the newtrackon locally with:

docker run -p 8080:8080 corralpeltzer/newtrackon --address=0.0.0.0 --ignore-ipv6

Until I got the error:

And I checked the logs:

2024-12-19 10:15:13,448 [INFO] [newtrackon] Updating udp://tracker.torrust-demo.com:6969/announce
2024-12-19 10:15:13,476 [INFO] [newtrackon] Tracker udp://tracker.torrust-demo.com:6969/announce sets protocol and port preferences from BEP34: [('udp', 6969), ('tcp', 443)]
2024-12-19 10:15:15,112 [INFO] [newtrackon] udp://tracker.torrust-demo.com:6969/announce Scraping UDP
2024-12-19 10:15:25,148 [INFO] [newtrackon] udp://tracker.torrust-demo.com:6969/announce status is DOWN. Cause: UDP timeout

It's a timeout. I will open a new issue on the tracker to a dd the new metric upd_average_response_latency

[josecelano]

The new issue to add the new metric upd_average_response_latency: torrust/torrust-tracker#1145

[josecelano]

I think the timeout in newtrackon for UDP announce requests is 10 seconds. See https://raw.githubusercontent.com/CorralPeltzer/newTrackon/c5af5c3d0e5d0179f9394f5a4492da8eca0282f3/newTrackon/scraper.py

I've deployed the new tracker version with new metrics. For the time being (20 minutes after restarting the tracker), the average processing time for announce requests is below 400 milliseconds. However that time is only backed processing, it does not include network latency sending the response back to the client.

[josecelano]

According to newtrackon data the latest response time is 180 ms.

[josecelano]

If the processing time is low, I guess that would mean, the requests are not even reaching the application (the "receiver"). Maybe we could try to fine tune the operating system.

ChatGPT suggestion

The number of UDP requests a socket can accept on a Linux system (like Ubuntu on a DigitalOcean droplet) is influenced by several system-level parameters. While UDP is connectionless and doesn't maintain queues for established connections like TCP, the system maintains buffers for receiving packets.

Here are the key factors:

---

1. Socket Receive Buffer Size (SO_RCVBUF)

• Description: Determines how much data the kernel will buffer for a UDP socket.
• Default: Varies by system; typically in the range of 212992 bytes (208 KB).
• Effect: If the buffer is full because packets are not being processed quickly enough, incoming packets are dropped.
• How to Check:

  cat /proc/sys/net/core/rmem_default

• How to Change:
  • Temporarily:

    sysctl -w net.core.rmem_default=262144  # Set to 256 KB, for example

  • Permanently:
    Add the following to /etc/sysctl.conf:

    net.core.rmem_default=262144
    

    Apply with:

    sysctl -p

---

2. Maximum Receive Buffer Size (rmem_max)

• Description: The upper limit for SO_RCVBUF.
• Default: Typically 212992 bytes (208 KB), but can be increased up to several megabytes.
• How to Check:

  cat /proc/sys/net/core/rmem_max

• How to Change:
  Similar to rmem_default.

---

3. UDP Memory Limits (net.ipv4.udp_mem)

• Description: Controls how much memory is allocated for all UDP sockets combined.
• How to Check:

  cat /proc/sys/net/ipv4/udp_mem

  This outputs three values (in memory pages):
  • Min: The minimum memory reserved for UDP sockets.
  • Pressure: When the system starts pressuring UDP sockets to release memory.
  • Max: The maximum memory that can be allocated.
• How to Change:
  • Temporarily:

    sysctl -w net.ipv4.udp_mem="4096 87380 16777216"

  • Permanently:
    Add the following to /etc/sysctl.conf:

    net.ipv4.udp_mem=4096 87380 16777216
    

---

4. File Descriptor Limits

• Description: The number of open file descriptors (sockets) a process can have.
• How to Check:

  ulimit -n

• How to Increase:
  • Temporarily:

    ulimit -n 65535

  • Permanently:
    Edit /etc/security/limits.conf:

    *    soft    nofile  65535
    *    hard    nofile  65535
    

---

5. Network Queue Limits

• Description: The kernel’s backlog queue for packets waiting to be processed.
• Relevant Parameters:
  • net.core.netdev_max_backlog: The maximum number of packets in the backlog before packets are dropped.
    • Default: Typically 1000 packets.
    • How to Change:

      sysctl -w net.core.netdev_max_backlog=5000

---

Summary:

• Default Limits: A typical Ubuntu configuration on DigitalOcean has moderate limits, such as a rmem_default of ~208 KB, which allows the socket to buffer around 140 UDP packets of 1500 bytes each.
• Scaling Recommendations:
  • Increase SO_RCVBUF and rmem_max for larger per-socket buffers.
  • Ensure udp_mem has a high enough maximum to handle many sockets.
  • Raise netdev_max_backlog if the network interface queue is dropping packets.

---

Practical Example:

For a DigitalOcean droplet handling high UDP traffic, you can configure:

• net.core.rmem_default=262144
• net.core.rmem_max=8388608
• net.ipv4.udp_mem=4096 87380 16777216
• net.core.netdev_max_backlog=5000

This allows a single UDP socket to buffer many more packets, assuming sufficient application performance to process them in time.

[josecelano]

Announce processing time looks fine for the last 12 hours:

[josecelano]

netstat -su
IcmpMsg:
    InType3: 49256
    InType5: 205
    InType11: 20
    OutType3: 746124
Udp:
    8187249 packets received
    1007279 packets to unknown port received
    0 packet receive errors
    8344788 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
IpExt:
    OutMcastPkts: 20
    InOctets: 425726197318
    OutOctets: 862919782235
    OutMcastOctets: 800
    InNoECTPkts: 4846361527
    InECT1Pkts: 37809
    InECT0Pkts: 300510
    InCEPkts: 17275
MPTcpExt:

From the output of netstat -su, here are the key observations regarding your UDP tracker and general UDP traffic:

---

Key Metrics

UDP Statistics

• Received Packets (packets received):
  • Total number of UDP packets received: 8,187,249.
• Packets to Unknown Port (packets to unknown port received):
  • 1,007,279 packets were sent to a port where no process was listening. This means clients are sending packets to closed ports. These packets are dropped.
• Packet Receive Errors (packet receive errors):
  • 0 indicates no packets were discarded due to malformed headers or other issues.
• Receive Buffer Errors (receive buffer errors):
  • 0 indicates no packets were dropped because the socket buffer was full.
• Packets Sent (packets sent):
  • 8,344,788 packets were successfully sent.

---

Key Insights

1. High Number of Packets to Unknown Ports:

   • A significant number of packets (1,007,279) were sent to ports where no application is listening. This could happen if:
     • Clients are misconfigured or attempting to contact an incorrect tracker port.
     • The tracker server was temporarily down or unreachable when clients attempted to contact it.
   • These packets are silently dropped.

2. No Buffer Errors:

   • The absence of receive and send buffer errors suggests that your server's UDP buffer sizes are adequate for the current load.

3. No Packet Receive Errors:

   • This indicates that no packets were dropped due to corruption or invalid headers.

4. Healthy Traffic Volumes:

   • The ratio of packets received (8,187,249) to sent (8,344,788) aligns with the typical behavior of a UDP tracker: sending responses to incoming requests.

---

[josecelano]

After deploying a new version, it has increased:

The main change was to use broadcast channels for metrics.

[josecelano]

Hi @da2ce7, I don't know why, but the uptime in newtrackon has increased:

If we go over the 95% uptime, the tracker could be included in the newtrackon list.

Grafana dashboard for the last 2 weeks:

[josecelano]

Hi @da2ce7, I don't know why, but the uptime in newtrackon has increased:

If we go over the 95% uptime, the tracker could be included in the newtrackon list.

Grafana dashboard for the last 2 weeks:

Maybe because I've increased the IP ban from 1h to 24h. newtrackon is sending the proper connection ID. Basically, the tracker is now serving better the clients that properly implementing the UDP protocol. The load has decreased and there are fewer timeouts.

[josecelano]

I run the newtrackon locally with:

docker run -p 8080:8080 corralpeltzer/newtrackon --address=0.0.0.0 --ignore-ipv6

Until I got the error:

And I checked the logs:

2024-12-19 10:15:13,448 [INFO] [newtrackon] Updating udp://tracker.torrust-demo.com:6969/announce
2024-12-19 10:15:13,476 [INFO] [newtrackon] Tracker udp://tracker.torrust-demo.com:6969/announce sets protocol and port preferences from BEP34: [('udp', 6969), ('tcp', 443)]
2024-12-19 10:15:15,112 [INFO] [newtrackon] udp://tracker.torrust-demo.com:6969/announce Scraping UDP
2024-12-19 10:15:25,148 [INFO] [newtrackon] udp://tracker.torrust-demo.com:6969/announce status is DOWN. Cause: UDP timeout

It's a timeout. I will open a new issue on the tracker to a dd the new metric upd_average_response_latency

☝ That was not needed. You can read the raw data in newtrackon:

https://newtrackon.com/raw