feat: updating L2 Block encoding and Rollup.process function

[benesjan]

Fixes #3936
Fixes #4010
Fixes #4011

[AztecBot]

Benchmark results

Metrics with a significant change:

• note_trial_decrypting_time_in_ms (32): 120 (+89%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit fe4538b3 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,380	179,492	716,036
l1_rollup_calldata_gas	221,132	866,264	3,447,836
l1_rollup_execution_gas	934,659	3,969,857	24,795,439
l2_block_processing_time_in_ms	1,444 (+1%)	5,494 (+2%)	22,295 (+3%)
note_successful_decrypting_time_in_ms	322 (+4%)	990 (+6%)	3,522 (+6%)
note_trial_decrypting_time_in_ms	31.6 (+48%)	⚠️ 120 (+89%)	146 (-1%)
l2_block_building_time_in_ms	16,353 (+1%)	64,762 (+1%)	259,613 (+1%)
l2_block_rollup_simulation_time_in_ms	11,889 (+1%)	47,139 (+1%)	189,403 (+1%)
l2_block_public_tx_process_time_in_ms	4,419 (+1%)	17,529 (+1%)	69,834 (+1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	16,011 (+1%)	29,960 (-2%)
note_history_successful_decrypting_time_in_ms	2,188 (-7%)	4,204
note_history_trial_decrypting_time_in_ms	122 (+29%)	183 (+6%)
node_database_size_in_bytes	3,580,115	3,647,895
pxe_database_size_in_bytes	29,923	59,478

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	213	44,681	25,561
private-kernel-ordering	166 (+1%)	43,753	14,809
base-rollup	1,195	128,338	881
root-rollup	79.7 (+1%)	4,088	677
private-kernel-inner	293	71,208	25,561
public-kernel-private-input	206	31,863	25,561
public-kernel-non-first-iteration	204	31,905	25,561
merge-rollup	7.80 (+3%)	2,608	881

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	2 leaves	8 leaves	16 leaves	32 leaves	128 leaves	64 leaves	512 leaves	1024 leaves	2048 leaves	8192 leaves
batch_insert_into_append_only_tree_16_depth_ms	12.8 (+2%)	22.5 (+24%)	13.7 (-25%)	20.2 (-1%)	27.2 (+12%)	71.1 (+7%)	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.9	17.5	23.0	31.6	47.0	143	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.634 (+1%)	0.656 (+6%)	0.551 (-9%)	0.553 (+3%)	0.489 (+1%)	0.444	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	N/A	N/A	N/A	93.5 (-2%)	53.5 (+3%)	254	475 (-1%)	933 (+2%)	3,592 (+1%)
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	N/A	N/A	N/A	159	96.0	543	1,055	2,079	8,223
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.507 (+2%)	0.506 (+2%)	0.442	0.435	0.433	0.428
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	N/A	N/A	N/A	145 (+10%)	64.1 (+3%)	383 (+1%)	757 (-2%)	1,489 (+2%)	6,060 (+3%)
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	N/A	N/A	N/A	197	104	691	1,363	2,707	10,771
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.567 (+9%)	0.507 (+1%)	0.480	0.480 (-2%)	0.481	0.486
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	68.8 (+2%)	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	94.1	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	0.599 (+1%)	N/A	N/A	N/A	N/A	N/A	N/A	N/A

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	15,455	32,219

Transaction processing duration by data writes.

Metric	0 new commitments	1 new commitments
tx_pxe_processing_time_ms	392 (+5%)	1,092 (+1%)

Metric	1 public data writes
tx_sequencer_processing_time_ms	538

[LHerskind]

Nice job. Got a couple of comments

[LHerskind]

If we keep track of the block number in the contract, e.g., for having an array of all the block archive etc, you can get away with not needing next available. Otherwise it might be possible to pack the number along with some of the other updatable values, e.g., timestamp or so.

[benesjan]

Yes, we could pack it with lastBlockTs and it would be quite nice to have the ability to easily fetch the latest block number from contract.

for having an array of all the block archive etc, you can get away with not needing next available

@LHerskind can you expand on this ^? I was thinking about just storing it as a normal storage value packed with lastBlockTs.

[LHerskind]

Recall when we were speaking about downloading a state snapshot such that you could sync really quickly.

So to make sure that the thing is valid, you have to ensure that it agrees with history, e.g., that the state actually matches the real state. The easiest way to do this (if you agree with the L1 contract for the canonical) you want to check that you agree with the contract. But if you are only having the last, you might have some troubles because it could have updated since the snapshot (this is very likely as we don't have state snapshots shared every block).
This means that you would either try to search back in blocks on L1 to find the matching or you should figure it out as you are searching for blocks.

Say that we are instead using something like below:

uint256 blockNumber
mapping(uint256 blockNumber => bytes32 archive) archives;

function process(...) {
  archives[blockNumber++] = archive;
}

Then it is almost instant to check that the snapshot is good, you just look up at that block and you are done.

In practice it is mainly a bit of performance diff as you can just lookup in state instead of having to look for the block though, so might not be worth it.

[benesjan]

We could modify the L2BlockProcessed event to emit not only block number but the archive as well. That would be cheaper and also very nice to use given that the blockNumber is indexed. WDYT?

(Will rework it to store the blockNumber in storage.)

[benesjan]

Realized that we have the block number in the header so we don't really need to store it in storage.

[LHerskind]

Not sure you can fully get away with not having it there. So, we have it there such that you can ensure the next block is the old + 1.

When doing the insertion into the archive, I guess you are matching the parent to be a position old, and then you ensure that old + 1 is empty and then insert.

Assuming that archives is filled with zeros. Then if jumping ahead, you need to make a header that is hashing to 0 in order to use it. And if going in the past, you fail because the old + 1 would not be 0.

Think the yellow paper also need a -1 actually in:

# Check that the parent is a valid parent
    assert merkle_inclusion(
        parent.hash(),
        parent_sibling_path,
        left.public_inputs.constants.global_variables.block_number,
        left.public_inputs.constants.last_archive.root
    )

It would mean that it fails due to a proof error instead of bad block so error message is less clear though.

[LHerskind]

As commented elsewhere. We need to constrain it in the circuits here instead of the contract.

[LHerskind]

Fix the formatting then I think its good 👍