rule 941160 using excessive cpu and time on some POST

[fischaz]

Hi team,

the team reported a slowness issue on the WAF when trying to POST a 600kb request to the server. After investigation, the Rule 941160 was found to be taking most of the time/cpu. it seems that the content of the POST request does not bode well with the regular expression.

some versions:
nginx(1.14.0)
modsecurity(eed6b5f8 2018-06-26 13:56:22 -0300)
modsecurity-nginx(37b76e8 2018-05-15 18:33:58 -0300)
owasp-modsecurity-crs(a216353 2018-01-02 10:35:32 -0300)

the rules were tested on both the 3.0 master and 3.1-dev branch.

Actual behavior:

• The POST eventually works but takes 30 seconds within the WAF before being sent to the application server.

Expected behavior:

• the POST should work withing 2 seconds for such a POST packet size to be validated and forwarded.

the POST is using a application/x-www-form-urlencoded POST with key/value pairs, except that the last value is 400kb (reallly taking most of it) and is URL encoded (no issue there). Unfortunately, once decoded, the value is actually a JSON object.

I'll attach the json object which I have sanitised (replaced words by xxx) but the behavior remained the same. I have just written a simple python and perl script to test the Regex in the CRS against the json text file and indeed, the script takes around 40 to 60 seconds to simply parse the regex and confirm there is no match.

Question: Is there an issue in the regular expression (a missing anchor or just too much recursion), or are we simply completely wrong to send a JSON object this way to the server and the web application need to be changed?

Sample python script to test the decoded JSON object against the rule 941160 regex

#!/usr/bin/python

import re
import sys
from urllib.parse import unquote


regex = "(?i)<[^\\w<>]*(?:[^<>\"'\\s]*:)?[^\\w<>]*(?:\\W*?s\\W*?c\\W*?r\\W*?i\\W*?p\\W*?t|\\W*?f\\W*?o\\W*?r\\W*?m|\\W*?s\\W*?t\\W*?y\\W*?l\\W*?e|\\W*?s\\W*?v\\W*?g|\\W*?m\\W*?a\\W*?r\\W*?q\\W*?u\\W*?e\\W*?e|(?:\\W*?l\\W*?i\\W*?n\\W*?k|\\W*?o\\W*?b\\W*?j\\W*?e\\W*?c\\W*?t|\\W*?e\\W*?m\\W*?b\\W*?e\\W*?d|\\W*?a\\W*?p\\W*?p\\W*?l\\W*?e\\W*?t|\\W*?p\\W*?a\\W*?r\\W*?a\\W*?m|\\W*?i?\\W*?f\\W*?r\\W*?a\\W*?m\\W*?e|\\W*?b\\W*?a\\W*?s\\W*?e|\\W*?b\\W*?o\\W*?d\\W*?y|\\W*?m\\W*?e\\W*?t\\W*?a|\\W*?i\\W*?m\\W*?a?\\W*?g\\W*?e?|\\W*?v\\W*?i\\W*?d\\W*?e\\W*?o|\\W*?a\\W*?u\\W*?d\\W*?i\\W*?o|\\W*?b\\W*?i\\W*?n\\W*?d\\W*?i\\W*?n\\W*?g\\W*?s|\\W*?s\\W*?e\\W*?t|\\W*?a\\W*?n\\W*?i\\W*?m\\W*?a\\W*?t\\W*?e)[^>\\w])|(?:<\\w[\\s\\S]*[\\s\\/]|['\"](?:[\\s\\S]*[\\s\\/])?)(?:formaction|style|background|src|lowsrc|ping|on(?:d(?:e(?:vice(?:(?:orienta|mo)tion|proximity|found|light)|livery(?:success|error)|activate)|r(?:ag(?:e(?:n(?:ter|d)|xit)|(?:gestur|leav)e|start|drop|over)?|op)|i(?:s(?:c(?:hargingtimechange|onnect(?:ing|ed))|abled)|aling)|ata(?:setc(?:omplete|hanged)|(?:availabl|chang)e|error)|urationchange|ownloading|blclick)|Moz(?:M(?:agnifyGesture(?:Update|Start)?|ouse(?:PixelScroll|Hittest))|S(?:wipeGesture(?:Update|Start|End)?|crolledAreaChanged)|(?:(?:Press)?TapGestur|BeforeResiz)e|EdgeUI(?:C(?:omplet|ancel)|Start)ed|RotateGesture(?:Update|Start)?|A(?:udioAvailable|fterPaint))|c(?:o(?:m(?:p(?:osition(?:update|start|end)|lete)|mand(?:update)?)|n(?:t(?:rolselect|extmenu)|nect(?:ing|ed))|py)|a(?:(?:llschang|ch)ed|nplay(?:through)?|rdstatechange)|h(?:(?:arging(?:time)?ch)?ange|ecking)|(?:fstate|ell)change|u(?:echange|t)|l(?:ick|ose))|m(?:o(?:z(?:pointerlock(?:change|error)|(?:orientation|time)change|fullscreen(?:change|error)|network(?:down|up)load)|use(?:(?:lea|mo)ve|o(?:ver|ut)|enter|wheel|down|up)|ve(?:start|end)?)|essage|ark)|s(?:t(?:a(?:t(?:uschanged|echange)|lled|rt)|k(?:sessione|comma)nd|op)|e(?:ek(?:complete|ing|ed)|(?:lec(?:tstar)?)?t|n(?:ding|t))|u(?:ccess|spend|bmit)|peech(?:start|end)|ound(?:start|end)|croll|how)|b(?:e(?:for(?:e(?:(?:scriptexecu|activa)te|u(?:nload|pdate)|p(?:aste|rint)|c(?:opy|ut)|editfocus)|deactivate)|gin(?:Event)?)|oun(?:dary|ce)|l(?:ocked|ur)|roadcast|usy)|a(?:n(?:imation(?:iteration|start|end)|tennastatechange)|fter(?:(?:scriptexecu|upda)te|print)|udio(?:process|start|end)|d(?:apteradded|dtrack)|ctivate|lerting|bort)|DOM(?:Node(?:Inserted(?:IntoDocument)?|Removed(?:FromDocument)?)|(?:CharacterData|Subtree)Modified|A(?:ttrModified|ctivate)|Focus(?:Out|In)|MouseScroll)|r(?:e(?:s(?:u(?:m(?:ing|e)|lt)|ize|et)|adystatechange|pea(?:tEven)?t|movetrack|trieving|ceived)|ow(?:s(?:inserted|delete)|e(?:nter|xit))|atechange)|p(?:op(?:up(?:hid(?:den|ing)|show(?:ing|n))|state)|a(?:ge(?:hide|show)|(?:st|us)e|int)|ro(?:pertychange|gress)|lay(?:ing)?)|t(?:ouch(?:(?:lea|mo)ve|en(?:ter|d)|cancel|start)|ime(?:update|out)|ransitionend|ext)|u(?:s(?:erproximity|sdreceived)|p(?:gradeneeded|dateready)|n(?:derflow|load))|f(?:o(?:rm(?:change|input)|cus(?:out|in)?)|i(?:lterchange|nish)|ailed)|l(?:o(?:ad(?:e(?:d(?:meta)?data|nd)|start)?|secapture)|evelchange|y)|g(?:amepad(?:(?:dis)?connected|button(?:down|up)|axismove)|et)|e(?:n(?:d(?:Event|ed)?|abled|ter)|rror(?:update)?|mptied|xit)|i(?:cc(?:cardlockerror|infochange)|n(?:coming|valid|put))|o(?:(?:(?:ff|n)lin|bsolet)e|verflow(?:changed)?|pen)|SVG(?:(?:Unl|L)oad|Resize|Scroll|Abort|Error|Zoom)|h(?:e(?:adphoneschange|l[dp])|ashchange|olding)|v(?:o(?:lum|ic)e|ersion)change|w(?:a(?:it|rn)ing|heel)|key(?:press|down|up)|(?:AppComman|Loa)d|no(?:update|match)|Request|zoom))[\\s\\x08]*?=";

r2 = re.compile(regex)
filename = sys.argv[1];
with open(filename, 'r') as file_fd:
        for line in file_fd:
                line = line.strip()
                m = r2.search(line)
                if m:
                        print("Value Matched regex\n")
                else:
                        print("No match")

[fischaz]

SO-9817_data.public.txt

[fischaz]

$ time python test_SO-9817.py SO-9817_data.public.txt 
No match

real	7m54.563s
user	7m53.967s
sys	0m0.041s

[csanders-git]

My inital thought was that this was do to libmodsec some how so i tested it on Apache with modsec 2.9.x

Here was my test harness (clearly i'm lazy and hacked this together)

import time
import requests

  f = open('Downloads/SO-9817_data.public.txt','r')
  content = f.read()
  payload = {'x' : content}
  values = []

  for i in range(0,100):
      start = time.time()
      r = requests.post("http://localhost", data=payload)
      end = time.time()
      values.append(end-start)

  total = 0
  for i in values:
      total += i
  avg = total/(len(values) * 1.0)

  print avg

The result was 0.603922491074, this was run on the owasp/modsecurity docker image so you can expect that it wasn't a particularly performant box.

As a result i'm going to close this from the CRS side. The regex itself has 25 steps which isn't super efficient but it is made up of a number of different rules. Feel free to take this up with the modsec v3 team and i'm frequently on that side commenting on things as well.

[fischaz]

Hi,

I appreciate the time you took and check the libmodsecurity. Would you be able to confirm that the regex that I ran in my python script (which should be copy/paste from the CRS rules) against the public file?

why is this one regular expression taking so long then?, is the libmodsecurity using re.find() or re.match() (in their C equivalent)?

I understand that this is now outside your project, but it would be great if you could just try to run y python script and see what I'm doing is wrong?

Many thanks,
J.

[csanders-git]

I'm always here to support @fischaz. Turns out modsec uses pcre where as Python uses a different regex engine. This may be the origin of your problem

[jleproust]

Hi,

I've found exactly the same problem, triggered by the same rule. The regexp takes several minutes of pure CPU time, matching a simple 800kB json payload:
[{"xxx": "2019-02-07T17:16:03", "xxxyyyy": "2019-02-07T17:16:01", "aaaayyyy": "2019-02-07T17:16:00", "abcde": "268447744", "kalsjdh": "F", "klasjdvhas": "268622082", "aaaa": "2019-02-07T17:16:02"}, {"xxx": "2019-02-07T17:16:04", "xxxyyyy": "2019-02-07T17:16:02", "aaaayyyy": "2019-02-07T17:16:01", "abcde": "268447745", "kalsjdh": "F", "klasjdvhas": "268622083", "aaaa": "2019-02-07T17:16:03"},...]

The payload is an array of thousands of identical dicts, with the same keys and similar values.

I'm running nginx 1.14 with libmodsecurity 1.0.0, modsecurity 3.0.3, crs 3.1.0, libpcre3 2:8.39-3 (Debian 9).

This is probably a pathological coincidence specific to the regexp implementation. Is there any way to mitigate this, short of disabling the rule?

[jleproust]

By the way 941170 has the same issue, though using a little less cpu than 941160 on similar payloads. I ended up disabling both to avoid a DoS vulnerability.