LMDB installed perfectly but its not showing up in working #2240
Comments
|
Hi @nasirbas1, Please make sure that nignx/connector is using the library that you have compiled with lmdb support. It is very likely that your webserver/connector is using the old library. |
|
@zimmerle for the leakage part will requests from blocked IP's stop leaking through with lmdb because with smdb even after hours of blocking some of the request still go through. |
|
@nasirbas1 the manner to block the request (timeout, number of attempts, block time, etc...) should consider that the counter increment is not atomic. There may be a difference from multiprocess sync. Regardless, from my experience, the lmdb sync is quite fast. 1 hour without a sync, sounds to me that you are still using the memory collection still. Regardless, if your intention is to block a given IP address, there are different manners to achieve such functionality in an atomic manner, such us: rbl |
|
@zimmerle I don't thinks this is the case , I removed all the symlinks and folders , basically recompiled everything and its still not working . I think it has taken up the storage mode , because earlier with smdb, atleast it used to set the variable on IP collection but now setvar does happen, but when I try to retrieve the value in next request its blank . There seems to be no place to view exactly whats going wrong ? |
|
you can enable ModSecurity debug logs. You will see the actual value of each variable upon resolution time. you can also set the numbers of workers to 1 to test if the problem is within the workers sync. |
|
@zimmerle So I have already turned the debug logs at level 9. Its correctly setting the variable , but I think its not able to persist it to the lmdb because on the very next request the value previously set in the variable is blank .... The same works perfectly, when I compile it with smdb . There seems to be no entry in the logs related to the persistance of values to lmdb |
|
if you don't have multiple workers nor restart the webserver the value should be persistent regardless of the collection backend. Can you share the rules that you are using? |
|
@zimmerle I do have multiple workers Please find below the rules Initialize IP Collection using the IP address which comes as True-IP from load-balancer SecRule REQUEST_HEADERS:True-IP "^\b(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b" "id:660001,phase:1,t:none,pass,log,capture,setvar:tx.client_ip=%{tx.1}" logic to block any ip basically sets IP.blocked = 2 SecRule REQUEST_URI "@contains /ip/blacklist" Checks if allowed variable on IP collections for this IP is 2 then drops this request SecRule IP:ALLOWED "@eq 2" "id:880001,phase:1,t:none,drop,log,logdata:'Dynamic Blacklist'" |
|
What is the patch to your lmdb file? is it consistent? can you keep only a single worker for testing? |
|
SecDataDir /etc/nginx/modsec/storage/ Have added all the permissions , so the user has all the permissions to write to this folder If you meant version its 2.9.1 + LMDB ....found v2.9.1 P.S. Apart from ./configure --with-lmdb & SecDataDir /etc/nginx/modsec/storage/ , do we need to do anything else to turn on lmdb support? |
|
please check the content and size of the files.... |
|
@zimmerle which files ... the ones which lmdb is supposed to create ? |
|
So, the support to lmdb does not seems to be enabled. let's figure out why.... Please paste the results of: (change the path to nginx binary to match to your installation) |
|
ldd /usr/local/nginx/sbin/nginx returned this [root@ip-10-0-1-242 sbin]# ldd /usr/local/nginx/sbin/nginx grep mod didnt have anything to return . |
|
how have you compiled the connector? |
|
@zimmerle I didn't find any instructions to compile the connector anywhere Step 1 -> Installing dependencies Step 2 -> Compiling Modsecurity & LMDB LMDB Modsecurity Step 3 -> Building nginx with libmodsecurity Step 4 -> Copy the modsecurity .so file Is there anything we have missed anywhere , because specifically compiling the connector was not present anywhere? |
|
I think that's a wrong way, LMDB was linked to libmodsecurity, not for the Nginx or the Nginx module. You can check that is linked or not with this command: and you have to see something like this: If the output of the command is empty, that means lmdb wasn't linked to libmodsecurity. |
|
@airween How is the linking to be done ? |
|
You don't need to care with linking, if you pass the What's the output of the command above? |
|
@airween I did run the configure script with lmdb flag and this is the complete output checking for a BSD-compatible install... /bin/install -c ModSecurity - v3.0.4 for Linux Mandatory dependencies
Optional dependencies
Other Options
|
|
Please, run this command, and share with us the result: I assume your path is |
|
ldd /usr/local/modsecurity/lib/libmodsecurity.so | grep lmdb liblmdb.so.0.0.0 => /lib64/liblmdb.so.0.0.0 (0x00007f40147b3000) |
|
Right, this means that the lmdb was successfully linked to your library. Have you seen the |
|
@airween The command dumped following output /etc/nginx/production/modsec-shared-collections |
|
Well, that's a very impressive list :). |
|
@airween , So I renamed all the modsec-shared-collections and modsec-shared-collections-lock |
|
If the If it doesn't work that means your rules aren't doing what you expect. |
|
@airween What is the relation between modsec-shared-collections and LMDB storage ? I saw ALSO Observations after running requests 1️⃣ On every such request which reads values from collection, the modsec-shared-collections-lock files gets updated (date changes). 2️⃣ On requests where we write to the collection (actually where we set IP.BLOCKED=1 ), both the 3️⃣ But Immediately after setting BLOCKED flag on the IP to some value , I am unable to read the variable value from the collection for next requests from the same IP . Its blank . Also the rules that we are using are as under Initialize IP Collection using the IP address which comes as True-IP from load-balancer SecRule REQUEST_HEADERS:True-IP "^\b(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b" "id:660001,phase:1,t:none,pass,log,capture,setvar:tx.client_ip=%{tx.1}" Logic to block any ip basically sets IP.blocked = 2 SecRule REQUEST_URI "@contains /ip/blacklist" "id:440001,chain,phase:1,t:none,deny,log,status:200" Checks if allowed variable on IP collections for this IP is 2 then drops this request SecRule IP:allowed "@eq 2" "id:880001,phase:1,t:none,drop,log,logdata:'Dynamic Blacklist'" |
|
hi @nasirbas1
LMDB stores the datas (key:value pairs) in modsec-shared-collections file. Here is a small tool, which can help you to dump its content: https://gist.github.com/airween/09a9c8dd033eaf5fa317bc6da07c64a4 To compile see the comment before the included headers. Then you can use:
and does those files exists?
correct.
correct.
now I don't see the reason, I have to check it - I'll try to do that later. |
|
@airween No the ip.pag and ip.dir files don't exist on the system. UPDATE 1 : @airween Output of the command is as under ./pgm/lmdbread modsec-shared-collections |
|
UPDATE 2 ip.allowed = 2 But while reading we were reading it as SecRule IP:ALLOWED "@eq 2" "id:880001,phase:1,t:none,drop,log,logdata:'Dynamic Blacklist'" So apparently Modsec rules are _ Case Sensitive _ 😝 P.S. Apparently I am still wondering how everything is working well, when |
|
hi @nasirbas1,
I think this is a bug, and there is a fix for that.
because you built the modsecurity as module. |
|
@airween Yeah thats pretty much it . We added modsecurity as a dynamic module hence the empty value. Nice thread for anyone in future mingling around modsecurity persistence ✌️ |
|
Right - feel free to close the issue, if you think you don't have more question (for this topic :)) |
|
@airween Already closed 👍 |
|
thanks @airween for the lmdb reader gist. I found using it deadlocks after initial run, I had to change the order of cursor/dbi/txn/env close. I am using LMDB 0.9.24: (July 24, 2019). this works for me without deadlocks - https://gist.github.com/mmelo-yottaa/8504a5bfc1d97c00c9015dae88159333. For the curious, this was the deadlock: NGINX+MODSEC: Program received signal SIGINT, Interrupt. LMDBREAD: __pthread_mutex_lock_full (mutex=0x7ffff7ff7040) at ../nptl/pthread_mutex_lock.c:311 |
oh, I just created that small tool to read the filled collection database while I used libmodsecurity's regression test. Thank you for your notification. Anyway, @defanator also has a fix to avoid the deadlock. |
|
@airween isn't this utility which is part of |
Perhaps it is. I've never tested yet. |
Hi @zimmerle ,
For persistant collections, we recompiled modsecurity with lmdb flag after installing lmdb,
and the installation worked perfectly fine and lmdb showed up in the installed packages after
running the
./configure --with-lmdb script
However lmdb is not working. Its not showing up in logs, nor are there any errors anywhere. Is there
any way to verify the installation .
Also for SMDB installation , given the fact that every worker process has its own memory which they sync up with smdb at intervals, if any ip is blocked (say I set IP.BLOCKED=1 and deny it after checking this BLOCKED flag) , why does it even after hours altogether(considering that all process are in sync by then) let some of the requests from that blocked IP to go through ??
OS Version : CentOS Linux 7
Nginx Version : 1.16.1
Modsecurity version : 3.03
Mandatory dependencies
Optional dependencies
-lGeoIP , -I/usr/include/
-lcurl , -DWITH_CURL
-lyajl , -DWITH_YAJL
+ LMDB ....found v0.9.22
-llmdb , -DWITH_LMDB
-lxml2 -lz -lm -ldl, -I/usr/include/libxml2 -DWITH_LIBXML2
The text was updated successfully, but these errors were encountered: