Avoid the puppet clj-http-client in benchmark to improve performance #3886
Assignees
Comments
rbrw
added a commit
to rbrw/puppetdb
that referenced
this issue
Oct 5, 2023
Some previous investigations have suggested that the puppet client has performance issues (at least the sync client), and recent pdb benchmarking has implied something similar -- it could only emulate about 80k hosts with a 30s run interval, no matter how many senders or simulation threads it had (on a 60-core, non-hyperthreaded server). We suspect the puppet client may be adding extra complexity by (if nothing else) emulating sync via its async client (rather than just using the Apache sync client directly), and since benchmark only needs to make synchronous POSTs from existing threads (the senders), switch to the straightforward JDK client. Preliminary testing was promising, and after fully implementing this, benchmark was able to emulate over 140k hosts on the server mentioned above, using the same randomization percentage of 100. Guessing that we might now be hitting a CPU constraint, we changed the randomization to 10 percent, and were able to emulate at least 200k hosts. Fixes: puppetlabs#3886
rbrw
added a commit
to rbrw/puppetdb
that referenced
this issue
Oct 10, 2023
Some previous investigations have suggested that the puppet client has performance issues (at least the sync client), and recent pdb benchmarking has implied something similar -- it could only emulate about 80k hosts with a 30s run interval, no matter how many senders or simulation threads it had (on a 60-core, non-hyperthreaded server). We suspect the puppet client may be adding extra complexity by (if nothing else) emulating sync via its async client (rather than just using the Apache sync client directly), and since benchmark only needs to make synchronous POSTs from existing threads (the senders), switch to the straightforward JDK client. Preliminary testing was promising, and after fully implementing this, benchmark was able to emulate over 140k hosts on the server mentioned above, using the same randomization percentage of 100. Guessing that we might now be hitting a CPU constraint, we changed the randomization to 10 percent, and were able to emulate at least 200k hosts. Fixes: puppetlabs#3886
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Just use the JDK client. Some previous investigations have suggested that the puppet client has performance issues (at least the sync client), and recent pdb benchmarking has implied something similar -- it could only emulate about 80k hosts with a 30s run interval, no matter how many senders or similation threads it had (on a large 32-core server).
Since benchmark only needs to make synchronous POSTs from existing threads (the senders), the JDK's built in client should work fine, and preliminary tests have been promising.
We'll need to continue to support ssl-certs via pem files, but we can use puppet's
ssl-utilsfor that.The text was updated successfully, but these errors were encountered: