proxy abuse detection failing under 2.8.0 and nginx 1.4.x, 1.6.0

[aaccomazzi]

I have been using the following rules for detecting proxy abuse under apache and modsecurity 2.7.1 for a long time: https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/master/experimental_rules/modsecurity_crs_11_proxy_abuse.conf

The rules rely on this geo dataset: http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

I would like to use them under nginx, but I can't get them to work under any of the recent versions of nginx (1.4.x and 1.6.0) and modsecurity (2.7.x and 2.8.0). A request which carries a bogus proxy and which should be stopped instead causes the connection to be dropped e.g.:

curl --header 'X-Forwarded-For: 1.2.3.4' --verbose http://localhost/

and this appears in the logs:
nginxlog> 2014/04/29 15:07:00 [alert] 2794#0: worker process 2805 exited on signal 11 (core dumped)

The modsecurity debug log shows this:
[29/Apr/2014:15:07:00 --0400] [/sid#158a5b8][rid#1576798][/cgi-bin/showme/foo][4] Recipe: Invoking rule 15c8f00; [file "/etc/nginx/modsecurity.d/activated_rules/modsecurity_crs_11_proxy_abuse.conf"] [line "26"].
[29/Apr/2014:15:07:00 --0400] [/sid#158a5b8][rid#1576798][/cgi-bin/showme/foo][5] Rule 15c8f00: SecRule "TX:0" "@geoLookup " "chain,setvar:tx.geo_x-forwarded-for=%{geo.country_code}"
[29/Apr/2014:15:07:00 --0400] [/sid#158a5b8][rid#1576798][/cgi-bin/showme/foo][4] Transformation completed in 0 usec.
[29/Apr/2014:15:07:00 --0400] [/sid#158a5b8][rid#1576798][/cgi-bin/showme/foo][4] Executing operator "geoLookup" with param "" against TX:0.

[zimmerle]

Hi @aaccomazzi, we are currently fixing some problems in our nginx version, it includes the way that it handles those proxy requests.

The development code is available here:
https://github.com/SpiderLabs/ModSecurity/tree/nginx_refactoring

Can you get it a test?

Here is what i got with this development version:

(zimmerle@zlinux)-(~)$ curl --header 'X-Forwarded-For: 1.2.3.4' --verbose http://localhost:8080/
* Adding handle: conn: 0x67fbd0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x67fbd0) send_pipe: 1, recv_pipe: 0
* About to connect() to localhost port 8080 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.32.0
> Host: localhost:8080
> Accept: */*
> X-Forwarded-For: 1.2.3.4
>
< HTTP/1.1 200 OK
* Server nginx/1.7.0 is not blacklisted
< Server: nginx/1.7.0
< Date: Tue, 29 Apr 2014 21:36:22 GMT
< Content-Type: text/html
< Content-Length: 91
< Connection: keep-alive
< Last-Modified: Thu, 24 Apr 2014 12:40:12 GMT
< ETag: "5b-4f7c9267a3430"
< Accept-Ranges: bytes
< Vary: Accept-Encoding
<
<li> <a href="/virtualbox/">VirtualBox</a>
<li> <a href="/buildbot/">Igloo's buildbot</a>

* Connection #0 to host localhost left intact

[aaccomazzi]

Hi @zimmerle, thanks for the quick reply. Unfortunately the refactored branch does not help (nginx 1.6.0, CentOS 6.4). To clarify, the problem only appears when I enable the rules in modsecurity_crs_11_proxy_abuse.conf quoted above, i.e. when the geoLookup operator is at work. I should also mention that it makes no difference whether I use the GeoLiteCity.dat or the GeoIP.dat file. This happens both for http as well as https (see below).

curl -k --verbose --header 'X-Forwarded-For: 1.2.3.4' https://localhost

• About to connect() to localhost port 443
• Trying 131.142.198.210... connected
• Connected to localhost (127.0.0.1) port 443
• successfully set certificate verify locations:
• CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
• SSLv2, Client hello (1):
  SSLv3, TLS handshake, Server hello (2):
  SSLv3, TLS handshake, CERT (11):
  SSLv3, TLS handshake, Server key exchange (12):
  SSLv3, TLS handshake, Server finished (14):
  SSLv3, TLS handshake, Client key exchange (16):
  SSLv3, TLS change cipher, Client hello (1):
  SSLv3, TLS handshake, Finished (20):
  SSLv3, TLS change cipher, Client hello (1):
  SSLv3, TLS handshake, Finished (20):
  SSL connection using DHE-RSA-AES256-SHA
• SSL certificate verify result: self signed certificate (18), continuing anyway.

  GET / HTTP/1.1
  User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
  Host: localhost
  Accept: /
  X-Forwarded-For: 1.2.3.4

• Empty reply from server
• Connection #0 to host localhost left intact
  curl: (52) Empty reply from server
• Closing connection #0
• SSLv3, TLS alert, Client hello (1):

[zimmerle]

Linking this issue to the issue #658 as it seems that the nature of the problem is same, thus the solution will be valid for both. The problem is the related to a mutex creation, while the platform is already using all the available ones.

Mutex is created:
https://github.com/SpiderLabs/ModSecurity/blob/nginx_refactoring/apache2/modsecurity.c#L143

@aaccomazzi Meanwhile you can try what i have suggested on this comment:
#658 (comment)

[aaccomazzi]

Hi @zimmerle I think you're on the right track. I have removed all nginx generated signals on our server, restarted the daemon and run a query against it to trigger the geoLookup modsecurity rule. This correctly generated the 403 response and left some semaphores with a weird owner id (see below).

Happy to try new patches when you have something else to test...

[root@adsdmz1 mod_security]# ipcs -s
------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         
0x00000000 8257538    4294967295 600        1         
0x00000000 8290307    4294967295 600        1         
0x00000000 8323076    4294967295 600        1         
0x00000000 8355845    4294967295 600        1         

[zimmerle]

No longer a concern in libModSecurity. Marking it as won't fix. Further information about libModSecurity available here:
https://github.com/SpiderLabs/ModSecurity/tree/v3/master