regex high memory utilisation

[aboudreault]

Summary

It was reported by some customers that in some circumstances, kong can have a rapid memory utilisation increase. In only 5-10 minutes, a node can grow to 15-20G of memory. This can happen when:

1. The kong configuration has a lot of regex URIs (some customers have 5k-10k.)
2. The internal openresty regex cache is too small (default is 1024)
3. The traffic on the node is relatively small

Steps To Reproduce

1. Start a Kong node and load a lot of routes with regex URIs (I can provide a kong-ee db dump privately)
2. Do a query to an unknown route and wait some minutes: while true; do curl -vk https://localhost:8443/doesntexist; done.

Behind the hood, everytime kong receives a query, it tries to match it against all existing regex uris: https://github.com/Kong/kong/blob/master/kong/router.lua#L1565
Since the route doesn't exist... it calls re_find on all of them, which makes the openresty regex module automatically tries to compile and cache it in its internal lrucache. At the same time, it replaces and removes many keys (pointers). However, the cdata memory (pcre) is not freed right away. This will happen eventually when the garbage collector is triggered and call the ffi GC finalizers. When the traffic is low... the garbage collector doesn't seem to react fast enough to avoid an important memory grow caused by the internal cdata. In some envs (like k8s), nodes are simply killed.

Resolution

The resolution to avoid this issue is to make sure most of the regex URIs can fit in the cache: increasing lua_regex_cache_max_entries.

Improvements

1. It would be useful to see a warning that the number of regex URIs is higher than the cache. Users could proactively modify their configuration. I have a tentative branch for this: https://github.com/Kong/kong/tree/feat/display-warning-regex-cache
2. Although this issue is not hit by most of users, it seems important enough to do something more than a warning or telling users to fix their regex cache size. Here are some potential solutions/ideas that I would like to discuss:

• lua-resty-lrucache: add an option to specify that the ffi finalizer should be called when evicting an item. This could also be a threshold that when X items are evicted in a Y time window, display a warning and trigger the GC.
• openresty.regex: maybe it would be possible to detect at this level that we are replacing too many items in the lrucache
• kong: since we know the number of regex_uris at this level, we could probably add some logic/threshold to disable the re_find o flag (compile) when we detect we are just iterating too many regex_uris: https://github.com/Kong/kong/blob/master/kong/router.lua#L1564
• kong: if the number of regex_uris is 2x (example) the size of the regex_cache, display a warning and disable compile-once.

[aboudreault]

@dndx @bungle I am available to discuss more about this, and of course working on the solution we will agree with.

[dndx]

Thanks @aboudreault for the detailed analysis, we will try to look into reproducing this and have a discussion about fixes.

[subnetmarco]

Is there any update on this issue?

[dndx]

@PaulFischer-Kong could you take a look and see if we can arrange some priority for this issue?

[hanshuebner]

Is this still an issue with Kong 3.x?

[hbagdi]

This has been fixed. Please upgrade to Kong 3.1.0 or above.
Commit: https://github.com/kong/kong/commit/7b384d6c42623dec7860378b76163b980f0ef702/