(PDB-5691) benchmark: space out command types

[austb]

To alleviate the certname row contention, this replaces the pipeline that sends each command separately to the submission pipeline with a go-loop that schedules the sending of catalogs and reports at a later time (5-15 seconds delay for each). This should more accurately simulate the actual behavior and allow more realistic performance testing.

Every command runs scf-storage/maybe-activate-node! which causes contention on the certname row when multiple commands of the same certname are processed simultaneously. Previously we generated a set of commands for a certname and submitted them all at once, so there was contention on most submissions from benchmark. This is not reflective of reality because a catalog is delayed by the time it takes puppetserver to compile the node's catalog after receiving the factset, and the report is delayed after catalog compilation by the time it takes the agent to apply it.

[austb]

Marked as don't merge until I complete the performance testing of this change.

It might be helpful to review this with whitespace changes off

[austb]

This change does not appear to have degraded performance.

In the performance test environment, benchmark is still able to send the 83.333 commands per second required to simulate 50,000 nodes.

Perf test results (locally)

before this change to submit 5000 nodes

real    6m28.132s
user    10m0.507s
sys     0m24.076s

after this change to submit 5000 nodes

real    7m10.615s
user    10m11.129s
sys     0m24.837s

We see a small increase in user CPU time, and a moderate increase in real time. The increase in real time is at least partially due to the fact that instead of sending all commands immediately they are scheduled to send within the next 30 seconds. So instead of ending at a high rate, we end at a slow rate as we submit the final commands.

Ending before change

Sending 45.717 messages/s (load equivalent to 27,429 nodes with a run interval of 30 minutes)
Sending 46.684 messages/s (load equivalent to 28,010 nodes with a run interval of 30 minutes)
Cleaning up temp files from "/tmp/pdb-bench-16280516017042729251"
Sending 44.737 messages/s (load equivalent to 26,842 nodes with a run interval of 30 minutes)
Finished cleaning up temp files

Ending after change

Cleaning up temp files from "/tmp/pdb-bench-9898774072576177076"
Sending 36.749 messages/s (load equivalent to 22,049 nodes with a run interval of 30 minutes)
Finished cleaning up temp files
Sending 24.922 messages/s (load equivalent to 14,953 nodes with a run interval of 30 minutes)
Sending 10.923 messages/s (load equivalent to 6,553 nodes with a run interval of 30 minutes)
Sending 3.723 messages/s (load equivalent to 2,233 nodes with a run interval of 30 minutes)