feat: txpool persistence

[alexghr]

This PR adds a new implementation of a TxPool based on the kv-store introduced in #3628. It also amends the p2p-client's bootstrap flow.

The p2p-client saves the block number its on to the database and restores it the next time it is started. The initial sync happens as before (it syncs to the the tip of the chain, but instead of syncing from 0, it syncs from the last known block number) and after that, it re-publishes any transactions in its TxPool that haven't been processed already.

Fix #3365

[PhilWindle]

Do we need to do this? Transactions are propagated as soon as they are received. Do they need to be re-published when we start up?

[alexghr]

The use case I was thinking of: we're the only P2P client on the network, we get a bunch of transactions submitted to us but there's no one on the network to transmit them to. We shut down and come back some time later, so we broadcast any transactions stored locally. WDYT?

[PhilWindle]

I think it's fine for now. We can re-visit later.

[alexghr]

🥲 the new TxPool implementation does not guarantee that it returns transactions in insertion order 🥲

[AztecBot]

Benchmark results

Metrics with a significant change:

• note_successful_decrypting_time_in_ms (32): 1,076 (-16%)
• note_trial_decrypting_time_in_ms (8): 108 (+78%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 4a1c0df7 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	223,056	868,112	3,449,792
l1_rollup_execution_gas	842,143	3,595,220	22,205,454
l2_block_processing_time_in_ms	1,341 (+1%)	5,282 (+4%)	21,105
note_successful_decrypting_time_in_ms	396 (+13%)	⚠️ 1,076 (-16%)	3,861 (-1%)
note_trial_decrypting_time_in_ms	⚠️ 108 (+78%)	38.0 (-4%)	140
l2_block_building_time_in_ms	13,776 (+1%)	53,813	217,467
l2_block_rollup_simulation_time_in_ms	10,199 (+1%)	39,765	160,964
l2_block_public_tx_process_time_in_ms	3,551	13,990	56,323

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	15,269 (-3%)	29,181 (-2%)
note_history_successful_decrypting_time_in_ms	2,546 (-6%)	4,774 (-7%)
note_history_trial_decrypting_time_in_ms	82.4 (+1%)	310 (+53%)
node_database_size_in_bytes	3,782,380 (-4%)	3,868,488 (-9%)
pxe_database_size_in_bytes	29,940	59,499

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	200	43,109	20,441
private-kernel-ordering	115	25,833	9,689
base-rollup	2,058 (-1%)	244,144	881
root-rollup	82.9 (-6%)	4,088	889
private-kernel-inner	261	64,516	20,441
public-kernel-private-input	172	25,203	20,441
public-kernel-non-first-iteration	171	25,245	20,441
merge-rollup	9.76 (-1%)	2,608	881

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	10,331	26,317