Frang limit: connection_burst does not work

[krizhanovsky]

Moved from tempesta-tech/tempesta-test#240 by @KonsKo

Frang limit connection_burst does not work properly.

Client: curl
backend: nginx

Curl request: '-Ikf -v <url> -H "Host: tempesta-tech.com:8765" -H "Connection: close"'

Tempesta:

 frang_limits {
      connection_rate 4;
      connection_burst 2;
  }

docs https://github.com/tempesta-tech/tempesta/wiki/HTTP-security#connection-level-limits

connection_rate - I checked with 5 requests delayed with 0.125 sec between them (for purpose to not reach connection_burst limit) and everything works as expected. I checked journalctl and got expected warning

connection_burst - I checked with 3 requests NOT delayed with 0.125 sec between them. I checked journalctl and DID NOT get expected warning

connection_burst - I checked with 5 requests NOT delayed with 0.125 sec between them. I checked journalctl and got warning related to connection_rate

I use same approach for request_rate and request_burst - everything works as expected.

[avbelov23]

The burst of connections is taken into account in the 125 ms quantum. Curl is quite heavy for this, so it doesn't seem to work. I made tests based on ab.

[RomanBelozerov]

When the connection_burst limit is exceeded.
Tempesta blocks request forwarding for 1.5 seconds, but does not reset connection.

Test to reproduce

Warning for 3 connection and config frang_limits {connection_burst 2; connection_rate 4;}:

[ 7797.712059] [tempesta fw] Warning: frang: new connections burst exceeded for 127.0.0.1: 3 (lim=2)
[ 7797.713852] [tempesta fw] Warning: frang: new connections burst exceeded for 127.0.0.1: 4 (lim=2)
[ 7797.922866] [tempesta fw] Warning: frang: new connections rate exceeded for 127.0.0.1: 5 (lim=4)
[ 7798.129681] [tempesta fw] Warning: frang: new connections rate exceeded for 127.0.0.1: 6 (lim=4)
[ 7798.567626] [tempesta fw] Warning: frang: new connections rate exceeded for 127.0.0.1: 7 (lim=4)

Expected:
[ 7797.712059] [tempesta fw] Warning: frang: new connections burst exceeded for 127.0.0.1: 3 (lim=2)

Wireshark

[krizhanovsky]

The pcap log contains encrypted data, so it's hard to say where is which HTTP requests and responses. I also can't locate the test: github shows https://github.com/tempesta-tech/tempesta-test/blob/f90df946f8d9394fd60c75e75ccfebc12124e015/t_frang/test_connection_rate_burst.py#L114 as in a nonexistent branch. So I couldn't test the test. Was the test removed from kk-test?

I doubt that deproxy is a reliable tool to test bursts. Burst limits work within 125ms frames, i.e. the test needs to send 2 requests consequently within 125ms and Tempesta FW must also process them within the time frame. I think curl, sending several requests in single command execution, would be a more reliable tool to trigger burst limit.

There is a misconception in the configuration:

frang_limits {
    connection_burst 2;
    connection_rate 4;
}
block_action attack reply;

block_action attack reply means that Tempesta FW should send an HTTP error code if it thinks that a client attacks it. However, if we do not allow a new connection due to the frang limits, then there is nowhere to send the HTTP error codes. All we can do is just constantly hit the limit and print the warning.

1.5s is probably the consequence of the history of size FRANG_FREQ = 8, i.e. the history is only for 0.125ms * 8 = 1s plus maybe 0.5s for the Python overhead. We thought that the main reason for the limit is DDoS mitigation and if a bot has to wait for 1s between bursts, then the attack is mitigated. Having that a browser can establish 6 connections immediately for HTTP/1 (and this could be a reasonable limit), then each bot can make 6 connections/s, which aren't look too serious. However, this is TBD.

[krizhanovsky]

I created #1749 to finally solve the problem with overwhelming log messages.

[krizhanovsky]

@RomanBelozerov BTW keep #1664 in mind - this might affect some of the tests from #265.

[RomanBelozerov]

I also can't locate the test: github shows https://github.com/tempesta-tech/tempesta-test/blob/f90df946f8d9394fd60c75e75ccfebc12124e015/t_frang/test_connection_rate_burst.py#L114 as in a nonexistent branch.

This is concurent_connection_frang branch. For config:

frang_limits {
    connection_burst 2;
    connection_rate 4;

    ip_block off;
}
block_action attack reply;

Frang documentation.

If the limit is reached, the Tempest FW resets a connection (send TCP RST segment).

connection_burst does not send RST. Test to reproduce
Tempesta forwarded request the third client to server after 1-1.5 sec. But we expected RST for third client and repeat request from client. That did not happen. See this test in wireshark.

135, 137, 141 - 3 clients send requests.
138, 142 - Tempesta forwarded requests to server for 2 clients
144, 148 - server responses
146, 150 - Tempesta responses
232 - Tempesta forwarded request to server for 3 client after 1 sec (unexpected). RST?
234 - server response

I also didn't see the RST flag in TCP packages.

[krizhanovsky]

Yeah, Tempesta FW learnt how to send RST just recently, in #1643. I created a new issue #1751 - could you please add a comment about connection_burst to #1751?