"Database is not open. Use: SecGeoLookupDb directive." after reload

[jptosso]

Describe the bug

I'm using mmdb to block countries from my website and it works fine until I reload my nginx server and I start getting the message "Database is not open. Use: SecGeoLookupDb directive." in the debug log and no country would be blocked anymore.

Logs and dumps

Before reload:

[...] [/] [4] (Rule: 20) Executing operator "GeoLookup against REMOTE_ADDR.
[...] [/] [9] Target value: "111.111.111.111" (Variable: REMOTE_ADDR)
[...] [/] [9] Matched vars updated.
[...] [/] [9] Saving msg: Block China IP address
[...] [/] [4] Rule returned 1.
[...] [/] [4] Executing chained rule.
[...] [/] [4] (Rule: 0) Executing operator "Within" with param "CN CL" Was: "" against GEO:COUNTRY_CODE.
[...] [/] [9] Target value: "CN" (Variable: GEO:COUNTRY_CODE)
[...] [/] [9] Matched vars updated.
[...] [/] [4] Rule returned 1.

After reload:

[...] [...] [4] (Rule: 20) Executing operator "GeoLookup against REMOTE_ADDR.
[...] [...] [9] Target value: "111.111.111.111" (Variable: REMOTE_ADDR)
[...] [...] [4] Database is not open. Use: SecGeoLookupDb directive.
[...] [...] [4] Rule returned 0.

ModSecurity config log:

ModSecurity - v3.0.3-56-g6d5198b1 for Linux
Mandatory dependencies

• libInjection ....v3.0.3-56-g6d5198b1
• SecLang tests ....6d5198b

Optional dependencies

• GeoIP/MaxMind ....found
  • (MaxMind) v1.3.2
    -lmaxminddb, -DWITH_MAXMIND -I/usr/local/include
  • (GeoIP) v1.6.12
    -lGeoIP, -I/usr/include/
• LibCURL ....found v7.58.0
  -lcurl, -DWITH_CURL_SSLVERSION_TLSv1_2 -DWITH_CURL
• YAJL ....found v2.1.0
  -lyajl, -DWITH_YAJL -I/usr/include/yajl
• LMDB ....not found
• LibXML2 ....found v2.9.4
  -lxml2, -I/usr/include/libxml2 -DWITH_LIBXML2
• SSDEEP ....not found
• LUA ....found v501
  -llua5.1 -L/usr/lib/x86_64-linux-gnu/, -DWITH_LUA -DWITH_LUA_5_1 -I/usr/include/lua5.1

Other Options

• Test Utilities ....enabled
• SecDebugLog ....enabled
• afl fuzzer ....disabled
• library examples ....enabled
• Building parser ....disabled
• Treating pm operations as critical section ....disabled

To Reproduce

Use the following configuration with your own country code and restart nginx.

SecGeoLookupDb /mnt/nginx/defaults/geoip/geoip.mmdb

SecAction
"id:19,
phase:1,
nolog,
pass,
t:none,
setvar:'tx.high_risk_country_codes=CN CL'"
SecRule REMOTE_ADDR "@geoLookup" "chain,id:20,drop,msg:'Block China IP address',phase:2"
SecRule GEO:COUNTRY_CODE "@Within %{tx.high_risk_country_codes}"

Test with your own country code and you should get a 403 forbidden error code, then remove your country from the collection, reload and try again, you should receive 200. Add your country code again to the collection and you should get a 200 status and the previous reported message in the logs as the geoip database was unloaded.
Or you can just create a disruptive rule with geoip and after testing it works, you can check if it works after a reload.

Expected behavior

Allow me to reload the geoip database without restarting the server.

Server:

• ModSecurity version (and connector): ModSecurity master (6d5198b), Connector master (d7101e1)
• WebServer: 1.15.19
• OS: 4.15.0-1021-aws Fixed: nginx INCS config on fedora #21-Ubuntu x86_64
• libmaxminddb: 1.3.2 (from source)
• Nginx flags: ./configure
  --with-http_ssl_module
  --add-module=/tmp/ModSecurity-nginx/
  --with-http_ssl_module
  --with-http_sub_module
  --with-http_realip_module
  --with-http_sub_module
  --with-http_gunzip_module
  --with-http_gzip_static_module

More info

It seems like Utils::GeoLookup::getInstance().cleanUp(); is being called and Utils::GeoLookup::setDataBase is not being called again, I'm trying to find out the problem over there.

Small workaround

You can "reload" nginx by renewing the process:

create new master and workers

pid=$(cat /usr/local/nginx/logs/nginx.pid)
sudo kill -USR2 $pid

gracefully kill the old workers

sudo kill -WINCH $pid

gracefully kill the old master

sudo kill -QUIT $pid

That would reload the whole engine and nginx without any downtime but using many resources.
Benchmarks for 1000 reloads and winchs in a 8 core 16gb memory server with just 100 rules:

#https://gist.github.com/jptosso/aa72ff932d3fad39614923bccb2d175a
ubuntu@server:~$ sudo bash /etc/test.sh
winch: 17 seconds
reload: 35 seconds

[victorhora]

I believe the issue is probably due to the fact that the call to GeoLookup::setDataBase is made only at the parser here: https://github.com/SpiderLabs/ModSecurity/blob/v3/master/src/parser/seclang-parser.cc#L2706-L2723

The leads me to believe that on a reload, when GeoLookup::evaluate eventually is called, GeoLookup::lookup is called directly without any prior call to GeoLookup::setDataBase. We might be missing an init() at the geo_lookup.cc operator.

That being said, I could only reproduce this bug when I had BOTH MaxMind and the legacy GeoIP libraries/db installed.
So I believe that a valid temporary workaround is having only one of the libraries.

[jptosso]

Tested your workaround and it does work:

ModSecurity - v3.0.3-56-g6d5198b1 for Linux

Mandatory dependencies

• libInjection ....v3.0.3-56-g6d5198b1
• SecLang tests ....6d5198b

Optional dependencies

• GeoIP/MaxMind ....found
  • (MaxMind) v1.3.2
    -lmaxminddb, -DWITH_MAXMIND -I/usr/local/include
• LibCURL ....found v7.58.0
  -lcurl, -DWITH_CURL_SSLVERSION_TLSv1_2 -DWITH_CURL
• YAJL ....found v2.1.0
  -lyajl, -DWITH_YAJL -I/usr/include/yajl
• LMDB ....not found
• LibXML2 ....found v2.9.4
  -lxml2, -I/usr/include/libxml2 -DWITH_LIBXML2
• SSDEEP ....not found
• LUA ....found v502
  -lluajit-5.1 -L/usr/lib/x86_64-linux-gnu/, -DWITH_LUA -DWITH_LUA_5_2 -I/usr/include/lua5.2

[victorhora]

Thanks for the feedback @asdfuken. Good to know that it works as intended this way :)

Regardless, we'll try to get it fixed prior to 3.0.4 release. Thanks for reporting!

[welljsjs]

This is still an issue for us, we're on e9dce44.

Sorry what exactly is the workaround @jptosso ?

[zimmerle]

Fix under QA here - https://travis-ci.org/github/SpiderLabs/ModSecurity/builds/743954613?utm_source=github_status&utm_medium=notification

[davand01]

From the looks of this ticket, this fix has not yet made it to libmodsecurity? It's still planned for 3.1.0? Is there an estimate when it will become part of a release?

[martinhsv]

Hello @davand01 ,

The SecGeoLookupDb configuration item does work in ModSecurity v3. I have used it myself moderately recently with:

SecGeoLookupDB /usr/share/GeoIP/GeoLite2-Country.mmdb

[davand01]

@martinhsv - it works for me too, however if there is a reload (not restart) of nginx this exact error persists.
This means that for any potential change in the configuration, we need to restart our pods running nginx-ingress w/ModSecurity enabled. With this fix, it seems as if we don't have the need for that workaround.

[martinhsv]

Hello @davand01 ,

I do not see this effect when performing an nginx reload either. However, reading through this closed ticket, I'll note that I only have MaxMind installed.

Per the comments from 2019, it looks like this issue was only reproducible if "BOTH MaxMind and the legacy GeoIP libraries/db installed".

Is that the case for you?

[Dr-Lazarus-V2]

Hello, I am running Nginx with modsecurity which has both "MaxMind and the legacy GeoIP libraries/db installed". I can still notice the bug that when I perform an nginx -s "reload", it no longer works. I would have to restart the container (docker) running the engine for it to work!

Has this bug been resolved?

[airween]

Hello, I am running Nginx with modsecurity which has both "MaxMind and the legacy GeoIP libraries/db installed". I can still notice the bug that when I perform an nginx -s "reload", it no longer works. I would have to restart the container (docker) running the engine for it to work!

Has this bug been resolved?

I'm afraid it hasn't. But I think it's rather a connector issue than libmodsecurity3 - we have to investigate it.

Thanks for this reminder!

[Dr-Lazarus-V2]

Would it be okay for me to raise this as a separate issue, or will the team be reopening the existing issue?

[airween]

I reopen this issue - thanks!