chore: e2e on spots [ci rebuild]

[charlielye]

We spent 21538 compute minutes in e2e jobs in the last 2 days.

Note that even in the reduced cost running on smallest docker instance, CCI makes up 2/3rds of the cost! Their machines are over 10x the cost of spot instance.

Not a huge deal today, but if we intend on continuing to scale our collection of e2e tests it could be problematic.
For reference, we've double our CCI costs every month for the last 3 months from ~$800 to now ~$5000.

(Separate issue, but still hopeful we should eventually be able to run an e2e test in 10-20 seconds rather than minutes).

Ignoring spot stuff and price, this PR makes an important step towards the correct build-system abstraction, i.e. build-system is responsible for:

• Building docker containers.
• Running containers, or compose files.
• Handles the conditionality (rebuilding only what needs to be rebuilt).
• Handles pulling of dependent containers to do builds or runs of compose.
• Handles launching builds/runs on spots if wanting bigger cheap machines or lower cost.
• Uploading processing benchmarks.

Specifically this PR introduces cond_run_compose and cond_spot_run_compose scripts for launching built containers from compose scripts.

We should not be calling custom project scripts from config.yml. We should run a container (or compose file), and the running of that container should run whatever script needs to be run. This takes us a step closer to the next phase, which code-generating config.yml from build-manifest.yml. Please try to fit within this pattern and let's discuss if it doesn't work for some reason.

The code generating is more than just a convenience, but reflects that we have an abstraction that can be defined as a simple set of rules about how to build/run/deploy. It will make it much easier for engineers not familiar with build-system to add projects and for "just-the-right-thing" to happen. If we manage to get to that point, it may become relatively painless to shift the entire CI over to github actions, if so inclined.

[AztecBot]

Benchmark results

Metrics with a significant change:

• note_trial_decrypting_time_in_ms (32): 79.0 (-33%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 87fa6213 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,996	868,100	3,449,624
l1_rollup_execution_gas	842,083	3,595,208	22,204,993
l2_block_processing_time_in_ms	2,002 (-14%)	7,527 (-14%)	29,695 (-14%)
note_successful_decrypting_time_in_ms	297 (+7%)	870 (+9%)	3,130 (+8%)
note_trial_decrypting_time_in_ms	100 (+23%)	⚠️ 79.0 (-33%)	132 (-4%)
l2_block_building_time_in_ms	13,175 (-10%)	52,240 (-10%)	207,583 (-10%)
l2_block_rollup_simulation_time_in_ms	11,851 (-11%)	46,973 (-11%)	186,692 (-11%)
l2_block_public_tx_process_time_in_ms	1,282 (+1%)	5,137 (+1%)	20,415 (-1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	21,227 (-15%)	41,912 (-14%)
note_history_successful_decrypting_time_in_ms	2,032 (+7%)	3,927 (+5%)
note_history_trial_decrypting_time_in_ms	122 (-1%)	143 (-3%)
node_database_size_in_bytes	1,628,215	1,095,081
pxe_database_size_in_bytes	27,188	54,187

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	72.4 (-12%)	61,697	18,841
private-kernel-ordering	43.3 (-15%)	24,233	8,089
base-rollup	1,768 (-15%)	656,309	811
root-rollup	75.5 (-16%)	4,072	1,097
private-kernel-inner	52.9 (-10%)	81,504	18,841
public-kernel-private-input	41.3 (-11%)	41,455	18,841
public-kernel-non-first-iteration	26.4 (-9%)	41,497	18,841
merge-rollup	0.928 (+16%)	2,592	873

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	8,723	27,240