Merge branch 'main' into doc-latest #3908
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some previous investigations have suggested that the puppet client has performance issues (at least the sync client), and recent pdb benchmarking has implied something similar -- it could only emulate about 80k hosts with a 30s run interval, no matter how many senders or simulation threads it had (on a 60-core, non-hyperthreaded server). We suspect the puppet client may be adding extra complexity by (if nothing else) emulating sync via its async client (rather than just using the Apache sync client directly), and since benchmark only needs to make synchronous POSTs from existing threads (the senders), switch to the straightforward JDK client. Preliminary testing was promising, and after fully implementing this, benchmark was able to emulate over 140k hosts on the server mentioned above, using the same randomization percentage of 100. Guessing that we might now be hitting a CPU constraint, we changed the randomization to 10 percent, and were able to emulate at least 200k hosts. Fixes: puppetlabs#3886
More accurately describe what it does in preparation for adding an option to adjust the simulator pool size too. Leave the old option as a deprecated alias.
Change the --senders default to 2 or half the available cores, whichever is larger. Testing with the new JDK client (where each sender is handling everything itself), found that with a 60 core (non-hyperthreaded) host, only 16 senders were needed to hit a maximum rate with the local PuppetDB/PostgreSQL hosts.
…/ 2) Add a --simulators option to allow customizing the simulation thread count (previously just 4). Set the default to 2 or half the available cores, whichever is larger. Of the values examined, testing the new JDK client on a 60 core (non-hyperthreaded) host reached the highest commands/sec rate with 16 senders and 16 simulators.
…to-jdk-client (PDB-5712) Move benchmark to JDK client
(PE-36861) Update ezbake to 2.5.5
Move the timeout and ssl otions to an option map in preparation for an option to select the post method (http client).
Change the ssl-opts arg to opts in preparation for passing a custom :post.
Use the new :post option to only post via the JDK client from benchmark because we use the submit-* functions in both pdb and pdbext tests, and those cause fips tests to fail with the current JDK client configuration, e.g.: actual: java.lang.InternalError: java.security.NoSuchAlgorithmException: Unable to invoke creator for DEFAULT: Default key/trust managers unavailable This should restore the puppet http client as of 6fc7da1 (just before the switch to the JDK client in dc2033c) as the default client for command submission.
(PDB-5712) Fix FIPS tests after benchmark switch to JDK client
Allows you to run two or more Benchmark instances in parallel, offsetting the generated cert numbers so that the commands don't collide in the database. Let's you test massively when number of simulated nodes exceeds what a single Benchmark can push.
(PDB-5708) update version for 8.2.0 release
(PDB-5708) add lein-exec to ci profile
starting in commit 71c7cd2d126f45bab2262e3ccee2ce0e4852dd24 Puppet will log which server the catalog is requested from.
PuppetDB 8 only needs to maintain compatibility with Puppet 7 and 8, so it will not test Puppet 6
…se-notes (PDB-5700) 7.15.0 release notes
…e-notes (PDB-5707) 8.2.0 release notes
* 7.x: (PDB-5700) 7.15.0 release notes
Merge branch '7.x'
* main: (31 commits) (PDB-5707) 8.2.0 release notes (PDB-5700) 7.15.0 release notes (maint) Remove unnecessary Puppet version checks (PUP-11899) update integration tests for new Puppet log message Version 8.2.1-SNAPSHOT Version 8.2.0 (PDB-5708) add lein-exec to ci profile (PDB-5708) update version for 8.2.0 release (maint) Update version to 7.15.1-SNAPSHOT (maint) Update version to 7.15.0 for release (PE-35601) Add a cert number --offset to Benchmark Set clj-parent=7.2.6 (PDB-5712) client: restore puppet client for command posts (PDB-5712) submit-facts/submit-catalog/submit-report: generalize opts (PDB-5712) submit-command-via-http!: select send function via :post (PDB-5712) submit-command-via-http!: switch to option map Set clj-parent=5.6.6 (PE-36861) Update ezbake to 2.5.5 Set clj-parent=7.2.5 Set clj-parent=5.6.5 ...
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(PDB-5707) 8.2.0 release notes
(PDB-5700) 7.15.0 release notes
(maint) Remove unnecessary Puppet version checks
(PUP-11899) update integration tests for new Puppet log message
Version 8.2.1-SNAPSHOT
Version 8.2.0
(PDB-5708) add lein-exec to ci profile
(PDB-5708) update version for 8.2.0 release
(maint) Update version to 7.15.1-SNAPSHOT
(maint) Update version to 7.15.0 for release
(PE-35601) Add a cert number --offset to Benchmark
Set clj-parent=7.2.6
(PDB-5712) client: restore puppet client for command posts
(PDB-5712) submit-facts/submit-catalog/submit-report: generalize opts
(PDB-5712) submit-command-via-http!: select send function via :post
(PDB-5712) submit-command-via-http!: switch to option map
Set clj-parent=5.6.6
(PE-36861) Update ezbake to 2.5.5
Set clj-parent=7.2.5
Set clj-parent=5.6.5