Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PQL parsing of or clauses can exhaust memory #3874

Closed
rbrw opened this issue Sep 12, 2023 · 1 comment
Closed

PQL parsing of or clauses can exhaust memory #3874

rbrw opened this issue Sep 12, 2023 · 1 comment

Comments

@rbrw
Copy link
Contributor

rbrw commented Sep 12, 2023

It looks like the current PQL parser, using instaparse, consumes exorbitant quantities of RAM when attempting to parse or clauses. In a recent test instance with 1G allocated to puppetdb's heap, this query was enough to exhaust the heap in ~3m:

reports {
  latest_report? = true and (
    certname = "foo22" or
    certname = "foo21" or
    certname = "foo20" or
    certname = "foo19" or
    certname = "foo18" or
    certname = "foo17" or
    certname = "foo16" or
    certname = "foo15" or
    certname = "foo14" or
    certname = "foo13" or
    certname = "foo12" or
    certname = "foo11" or
    certname = "foo10" or
    certname = "foo9" or
    certname = "foo8" or
    certname = "foo7" or
    certname = "foo6" or
    certname = "foo5" or
    certname = "foo4" or
    certname = "foo3" or
    certname = "foo2" or
    certname = "foo1"
  )
}

Calling it with 21 OR clauses, the query succeeded in 31s, with 20 in 7s, with 19 in 4s, and so on back down to <1 once you reach 16 clauses.

As a temporary measure, use in instead of a set of ors; or wrap the ors in parentheses, although in general an in will be faster on the postgres side once the query's been parsed and postgres is actually running it.

@rbrw
Copy link
Contributor Author

rbrw commented Sep 12, 2023

Fixed by c5e1f88

@rbrw rbrw closed this as completed Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant