Add two stage Nginx rate limit

[melroy89]

delay=200 means it will allow the first 200 requests without any delay. After 200 requests you will hit the delaying of the requests, which is set to 5 requests/second. If the client keep requesting requests these will still get queued. However, after 300 requests in the buffer (bust=300), Nginx will start rejecting requests.

• Setup two stage rate limiter in Nginx
• Only rate limit based on the "Regular requests"
• These rejection requests will be put to the error log: /var/log/nginx/mbin_error.log.

See also: https://blog.nginx.org/blog/rate-limiting-nginx

---

If you use Fail2Ban this error log will be matched as nginx_error_log variable. You could use the error log to actually ban bots/users who are exceeding the rate limit (this ban is triggered if there are 50 rejected requests, default is only 3, in the past 10 minutes dedicated for the same IP):

[nginx-limit-req]
enabled  = true
port     = http,https
action   = %(action_)s
logpath  = %(nginx_error_log)s
findtime = 10m
maxretry = 50
bantime  = 1w

I will later document Fail2Ban config in a different markdown file / docs.

[melroy89]

When you really spam your own server (don't do evil on other servers). The error log should show something like this eventually:

2024/12/24 23:14:02 [error] 611449#611449: *1310 limiting requests, excess: 300.365 by zone "mbin_limit", client: 192.168.1.102, server: kbin.melroy.org, request: "GET /media/cache/resolve/post_thumb/42/62/42621fe78949e6c9540cc07ea3c236f562bc7235994c501bd722b4faab0df8bb.jpg HTTP/3.0", referrer: "https://kbin.melroy.org/"
2024/12/24 23:14:02 [error] 611449#611449: *1311 limiting requests, excess: 300.360 by zone "mbin_limit", client: 192.168.1.102, server: kbin.melroy.org, request: "GET /media/cache/resolve/post_thumb/81/b4/81b40b10bc6809845690f6e63232c34129eb39ba15ec6699618aa98b8679bbfc.png HTTP/3.0", referrer: "https://kbin.melroy.org/m/unixporn@lemmy.ml/t/633624/New-setup-because-I-accidentally-nuked-my-last-one"
2024/12/24 23:14:03 [error] 611449#611449: *1315 limiting requests, excess: 300.815 by zone "mbin_limit", client: 192.168.1.102, server: kbin.melroy.org, request: "GET /media/cache/resolve/avatar_thumb/51/42/51423acaf2a0870e0ab1a4aee1fcd998cc14dcc083e59cd72bb4b048d52540df.gif HTTP/3.0", referrer: "https://kbin.melroy.org/m/unixporn@lemmy.ml/t/633624/New-setup-because-I-accidentally-nuked-my-last-one"
2024/12/24 23:14:03 [error] 611449#611449: *1316 limiting requests, excess: 300.815 by zone "mbin_limit", client: 192.168.1.102, server: kbin.melroy.org, request: "GET /media/cache/resolve/avatar_thumb/81/b4/81b40b10bc6809845690f6e63232c34129eb39ba15ec6699618aa98b8679bbfc.png HTTP/3.0", referrer: "https://kbin.melroy.org/m/unixporn@lemmy.ml/t/633624/New-setup-because-I-accidentally-nuked-my-last-one"
2024/12/24 23:14:03 [error] 611449#611449: *1317 limiting requests, excess: 300.815 by zone "mbin_limit", client: 192.168.1.102, server: kbin.melroy.org, request: "GET /media/cache/resolve/entry_thumb/1a/c6/1ac6d47b97f14c51b8c39642b107259bf1aeffdd4c8f5ce1e44fa26a9e9de4f0.png HTTP/3.0", referrer: "https://kbin.melroy.org/m/unixporn@lemmy.ml/t/633624/New-setup-because-I-accidentally-nuked-my-last-one"
2024/12/24 23:14:03 [error] 611449#611449: *1318 limiting requests, excess: 300.815 by zone "mbin_limit", client: 192.168.1.102, server: kbin.melroy.org, request: "GET /media/cache/resolve/avatar_thumb/0e/20/0e20a50c43a89ebd065863eb1d0dce7c5442846470fd92623ebccd6a199fc9a1.png HTTP/3.0", referrer: "https://kbin.melroy.org/m/unixporn@lemmy.ml/t/633624/New-setup-because-I-accidentally-nuked-my-last-one"
2024/12/24 23:14:03 [error] 611449#611449: *1319 limiting requests, excess: 300.815 by zone "mbin_limit", client: 192.168.1.102, server: kbin.melroy.org, request: "GET /media/cache/resolve/avatar_thumb/a2/a7/a2a729d1e632b538dd68164eb71add47d6b512e95bead98479376f52fec78890.png HTTP/3.0", referrer: "https://kbin.melroy.org/m/unixporn@lemmy.ml/t/633624/New-setup-because-I-accidentally-nuked-my-last-one"
2024/12/24 23:14:03 [error] 611449#611449: *1320 limiting requests, excess: 300.815 by zone "mbin_limit", client: 192.168.1.102, server: kbin.melroy.org, request: "GET /media/cache/resolve/avatar_thumb/3c/b7/3cb7274f8e19959547f81a3f1c34580e033560e4c72869daf7bd12bb1df6b0cd.png HTTP/3.0", referrer: "https://kbin.melroy.org/m/unixporn@lemmy.ml/t/633624/New-setup-because-I-accidentally-nuked-my-last-one"
2024/12/24 23:14:03 [error] 611449#611449: *1321 limiting requests, excess: 300.815 by zone "mbin_limit", client: 192.168.1.102, server: kbin.melroy.org, request: "GET /media/cache/resolve/avatar_thumb/b0/1e/b01e8509567bce602989303993a6503c2a38b955de10e1e9ee437c576ecbec8d.png HTTP/3.0", referrer: "https://

Again, if you configured Fail2ban to pick this up (eg. 50 of these limiting requests errors). You can auto-ban IPs for an x amount of time (eg. 1 day or 1 week).