CI: Speed up CircleCI builds by increasing parallelism and resource classes

[michaeldiamant]

Extends #4426 with configuration tuning to reduce build time in exchange for higher costs.

Here's an estimated before (master) / after view taken from https://docs.google.com/spreadsheets/d/13JM3vqOWFpzWrtI2hT02OpvojP2tdYm89GLR3aqY0t4/edit#gid=537120180.

	Before	After	Change %
Duration (min)	35	14.4	-59%
$ annually	$17,971	$115,891	545%

Notes:

• To compare configuration changes, I found it easier to use yq: yq '.jobs | del(.*.steps, .*.working_directory, .*.parameters, .*.docker, .*.environment)' .circleci/config.yml.
  • Aside: If someone has a cleaner query to show a similar view, I'd be interested in seeing it.
• I investigated other configurations not shown in the PR. Generally, I think the PR presents a local optimum and reasonable stopping point.
• I tried heavier levels of parallelism and ran into queueing issues. I can elaborate live, but my view is we've exhausted easy parallelism wins and we must look towards other avenues to materially improve the build.
  • Obvious other avenues include: faster round times, https://circleci.com/docs/parallelism-faster-jobs#splitting-by-timing-data.

[codecov]

Codecov Report

Merging #4427 (a311590) into master (e4d6d42) will increase coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4427      +/-   ##
==========================================
+ Coverage   55.19%   55.22%   +0.03%     
==========================================
  Files         398      398              
  Lines       50165    50165              
==========================================
+ Hits        27689    27706      +17     
+ Misses      20159    20138      -21     
- Partials     2317     2321       +4     

Impacted Files	Coverage Δ
logging/telemetryFilteredHook.go	86.66% <0.00%> (-3.34%)	⬇️
cmd/tealdbg/dryrunRequest.go	53.70% <0.00%> (-1.86%)	⬇️
util/metrics/counter.go	88.04% <0.00%> (-1.09%)	⬇️
network/requestTracker.go	70.25% <0.00%> (-0.87%)	⬇️
agreement/trace.go	85.12% <0.00%> (-0.70%)	⬇️
cmd/tealdbg/local.go	73.61% <0.00%> (-0.35%)	⬇️
tools/network/dnssec/testHarness.go	82.06% <0.00%> (-0.35%)	⬇️
data/bookkeeping/block.go	56.27% <0.00%> (-0.34%)	⬇️
crypto/merkletrie/cache.go	88.72% <0.00%> (-0.30%)	⬇️
agreement/player.go	96.32% <0.00%> (-0.29%)	⬇️
... and 10 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[michaeldiamant]

Closing in favor of #4448. Based on live discussion, opting for short-term increases in build duration instead of increasing costs.