Backend Response Crashing ModSecurity in Conjunction With Mod_mem_cache

[rcbarnett-zz]

MODSEC-164: A faulty backend is able to segfault a reverse proxy with modsecurity, SecResponseBodyAccess On and mod_mem_cache enabled.

Faulty response:
HTTP/1.1 301 Moved Permanently
Date: Mon, 14 Jun 2010 08:45:26 GMT
Server: netcat
Content-Type: text/html; charset=UTF-8
Content-Length: 10
Location: http://www.example.com/target
Connection: close

Note that the response announces a Response Body but none is transferred.

This is the apache error-log:

[Tue Jun 15 15:25:23 2010] [debug] mod_proxy_http.c(1732): proxy: start body send
[Tue Jun 15 15:25:27 2010] [debug] mod_proxy_http.c(1836): proxy: end body send
[Tue Jun 15 15:25:27 2010] [debug] proxy_util.c(2029): proxy: HTTP: has released connection for (127.0.0.1)
[Tue Jun 15 15:25:27 2010] [debug] mod_cache.c(705): cache: Caching url: /proxy
[Tue Jun 15 15:25:27 2010] [debug] mod_cache.c(711): cache: Removing CACHE_REMOVE_URL filter.
[Tue Jun 15 15:25:27 2010] [debug] mod_mem_cache.c(752): mem_cache: URL http://127.0.0.1:8000/proxy? didn't receive complete response, not caching
[Tue Jun 15 15:25:27 2010] [debug] mod_cache.c(912): (20014)Internal error: cache: store_body failed

2-3 seconds afterwards, the server process crashes. Note that the root process does not crash. But he is no longer able to replace a killed child process.

Details:

Apache compilation on debian:
$ ./configure --prefix=/data/apache-2.2.15 --enable-mods-shared=all --enable-proxy --enable-proxy-http --enable-cache --enable-mem-cache --with-included-apr

ModSecurity compilation on debian:
$ ./configure --with-apxs=/data/folinic/apache/apache-2.2.15/bin/apxs --with-apu=/data/apache-2.2.15/bin/apu-1-config

Apache config:
ServerName www.example.com
ServerRoot /data/apache-2.2.15
User www-data
Group www-data

PidFile /tmp/httpd.pid
LockFile /tmp/httpd.lock

Listen 127.0.0.1:8000

ErrorLog /tmp/error.log
LogLevel debug

LoadModule unique_id_module modules/mod_unique_id.so
LoadModule security2_module modules/mod_security2.so

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so

LoadModule cache_module modules/mod_cache.so
LoadModule mem_cache_module modules/mod_mem_cache.so

SecRuleEngine On
SecResponseBodyAccess On

CacheEnable mem /

<VirtualHost 127.0.0.1:8000>

ErrorLog /tmp/error.log
ProxyPass /proxy http://127.0.0.1:7030/

Backend Configuration:
/bin/nc -l -s 127.0.0.1 -p 7030 < /tmp/bogus_response
-> see above for the response and be sure to kill netcat by hand when the response has been written out (which is immediately after connection).

Request crashing the server:
curl -v http://127.0.0.1:8000/proxy

This bug does not appear with Apache 2.2.14. But there has been a lot of coding in mod_cache/mod_mem_cache between 2.2.14 and 2.2.15.
I am not sure this is a bug in ModSecurity. Note that disabling response body access in ModSecurity makes the error go away, while the
error message above persists. So Mod_mem_cache is able to deal with the bogus response, while ModSecurity seems to have apache crash.

[rcbarnett-zz]

Original reporter: dune73

[rcbarnett-zz]

brectanus: I'd like to get a bit more info on this...

1. Just to clarify, it does not happen with caching off?

2. Please add a backtrace from the core you generated as I cannot do much with the core given I do not have the same binaries. Something like:

% gdb /path/to/httpd /path/to/core
gdb> thread apply all bt full

I'll at least need the detailed output of the thread that crashed.

-B

[rcbarnett-zz]

dune73: Debian Core File

[rcbarnett-zz]

dune73: Hi Brian,

Thanks for the quick response.

1: Yes, commenting out the CacheEnable directive in the example config makes the error go away. Just like disabling response body access in ModSecurity makes it go away too. I guess it has to do with the interaction between the two modules.

2: Here is the output:

Core was generated by `/data/folinic/apache/apache-2.2.15/bin/httpd -f /tmp/folinic_httpd.conf -k star'.
Program terminated with signal 11, Segmentation fault.
[New process 14902]
#0 0x00007f6c9d9976ad in apr_brigade_cleanup (data=) at buckets/apr_brigade.c:44
44 apr_bucket_delete(e);
(gdb) thread apply all bt full

Thread 1 (process 14902):
#0 0x00007f6c9d9976ad in apr_brigade_cleanup (data=) at buckets/apr_brigade.c:44
b = (apr_bucket_brigade *) 0x72be60
#1 0x00007f6c9d552c2f in apr_pool_destroy (pool=0x7236e8) at memory/unix/apr_pools.c:2308
active =
allocator =
#2 0x00000000004470bc in ap_process_http_connection (c=0x7194f0) at http_core.c:199
r = (request_rec *) 0x723760
csd = (apr_socket_t *) 0x0
#3 0x0000000000443173 in ap_run_process_connection (c=0x7194f0) at connection.c:43
n = 0
rv = 7519848
#4 0x000000000044e7a0 in child_main (child_num_arg=) at prefork.c:662
current_conn =
csd = (void *) 0x719300
ptrans = (apr_pool_t *) 0x719288
allocator = (apr_allocator_t *) 0x6ba060
status =
i =
lr =
pollset = (apr_pollset_t *) 0x717418
sbh = (ap_sb_handle_t *) 0x717410
bucket_alloc = (apr_bucket_alloc_t *) 0x7216d8
last_poll_idx = 0
#5 0x000000000044ea64 in make_child (s=0x681848, slot=0) at prefork.c:758
No locals.
#6 0x000000000044f00c in ap_mpm_run (_pconf=, plog=,
s=) at prefork.c:776
index =
remaining_children_to_start = 5
rv =
#7 0x00000000004285e5 in main (argc=5, argv=0x7fffffffe4a8) at main.c:740
c = 102 'f'
configtestonly = 0
confname = 0x7fffffffee00 "/tmp/folinic_httpd.conf"
def_server_root = 0x458c10 "/data/folinic/apache/apache-2.2.15"
temp_error_log = 0x0
error =
process =
server_conf = (server_rec *) 0x681848
pglobal = (apr_pool_t *) 0x678128
pconf = (apr_pool_t *) 0x67a138
plog = (apr_pool_t *) 0x6ac2c8
ptemp = (apr_pool_t *) 0x67e158
pcommands = (apr_pool_t *) 0x67c148
opt = (apr_getopt_t *) 0x67c238
rv =
mod =
optarg = 0x7fffffffee00 "/tmp/folinic_httpd.conf"

If there is more debugging information that I can extract, then just let me know. I'll be glad to help.
I am working for this customer today and then again next week.

[rcbarnett-zz]

dune73: Hi Brian,

I'm only back on this job today.

This is the output of "httpd -V":

root@v04prv apache-2.2.15>./bin/httpd -V
Server version: Apache/2.2.15 (Unix)
Server built: Jun 15 2010 13:10:24
Server's Module Magic Number: 20051115:24
Server loaded: APR 1.4.2, APR-Util 1.3.9
Compiled using: APR 1.4.2, APR-Util 1.3.9
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/data/folinic/apache/apache-2.2.15"
-D SUEXEC_BIN="/data/folinic/apache/apache-2.2.15/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

Loading ModSec as the last modules does not make any difference.

Above, there should be all the infos to duplicate the environment. We encountered the bug in Solaris 10 and took great care to get a minimal Linux environment to reproduce the error. But I'll be glad to offer help.

[rcbarnett-zz]

brectanus: I am not sure it matters, but you may want to try APR/APRUtil both on 1.4 or 1.3 and not mix them. I am of town until July 8 or so, so will not be able to look at it until then.

[rcbarnett-zz]

brectanus: That, unfortunately did not give me much :) Couple things I'd like to clear up:

1. The crash seems to have occurred in the APR code (cleanup routine). Since you built this with the Apache included APR, please verify that Apache and ModSecurity are running against the correct APR (i.e. the Apache included and not a system installed version). Usually "/data/folinic/apache/apache-2.2.15/bin/apachectl -V" will give you this. You may have to force it with an LD_LIBRARY_PATH or similar in the startup script if it finds the system lib before the Apache included APR.

2. I am curious if you load ModSecurity as the last module if it makes any difference.

I'll see if I can duplicate this environment, but it will have to wait until tomorrow.

[rcbarnett-zz]

bpinto: Hi Christian,

Still having this issue ?

thanks

Breno

[rcbarnett-zz]

bpinto: Closing this now. It is a very old ticket with no recent feedback. Also it looks like an apr cleanup code problem or something related with different apr lib versions in apache / modsec