feat!: note type ids

[nventuro]

Closes #2820. Additional context and design discussion can be found here.

---

This PR introduces the notion of Note Type IDs, identifiers for each note type that are unique within a contract. Different contracts may share the same IDs, and even use the same ID for notes of different types. Note Type IDs are only meaningful when referring to a single contract.

I considered two ways to implement this. The first was to add a new field to NoteHeader and leave ID allocation up to the contract in a similar way storage slots are determined. The call to new for each state variable would populate this field and make it available down the call stack. This results in a number of issues, including creating a dependency on the note type id in places where we need the header but don't care about this new value (e.g. when hashing), and forcing us to change the new() signature to have an extra field (so that the macro will work nicely on all state variables), despite it being entirely unused in some contexts (such as public state variables, which require no note type id as there's no hashing/nullification).

An alternative approach (the one followed in this PR) is to make the note type id a property of the note's themselves via a trait function get_note_type_id() which returns a constant. This results in a massively smaller diff as we don't need to change anything about state variables, their new function nor note headers: we simply fetch the note type deep in the call stack right before calling the oracle, avoiding a lot of passing values around (I'd imagine this also leads to a smaller circuit, but I haven't actually measured it).

The only problem here is that due to the orphan rule, we can't have a contract implement an external trait (the one for get_note_type_id()) for an external type (the note): the implementation must be in either of these two modules. It is possible to alter the note module via macros (something like a derive macro, which we already want to automatically serialize and deserialize notes), but the current macros are not yet suitable to do this. @vezenovm is working on a related feature that lays some of the foundation to what we want to achieve here.

Eventually we'll be able to produce all of the impls for get_note_type_id by keeping track of a per-contract counter (essentially enumerating all imported note types - see #4519). For now, I simply provided unique hardcoded values which correspond to the name of the note type. Once this is done, implementing #2918 should be trivial. Other future work includes exporting these type ids as part of the contract artifact (#4518).

---

Something that is potentially missing here is an example of a contract that uses two maps with different underlying note types. I'd leave that for a separate PR to not pollute this one with a bunch of new examples and associated tests, since it's already relatively large and touches a large number of files.

---

Minor curiosity: the PXE service uses ExtendedNote both to add and return notes. For simplicity, I added the type id to this type, meaning we end up storing the type id in the local note database. This is not strictly required, but the impact is negligible and it results in easier debugging and a (much!) smaller diff.

[AztecBot]

Benchmark results

No metrics with a significant change found.

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit acf5cf23 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,956 (+1%)	181,604 (+1%)	724,196 (+1%)
l1_rollup_calldata_gas	227,192 (+2%)	888,740 (+2%)	3,535,832 (+2%)
l1_rollup_execution_gas	320,383 (+1%)	1,007,188 (+2%)	3,756,210 (+2%)
l2_block_processing_time_in_ms	1,152	4,350 (+1%)	17,254 (+1%)
note_successful_decrypting_time_in_ms	317 (+1%)	987 (+1%)	3,680 (+1%)
note_trial_decrypting_time_in_ms	28.8 (+14%)	113 (+6%)	141 (-4%)
l2_block_building_time_in_ms	18,383	72,523	291,095
l2_block_rollup_simulation_time_in_ms	13,505	53,262	213,938
l2_block_public_tx_process_time_in_ms	4,850	19,183	76,896

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	15,146 (+1%)	29,048 (+1%)
note_history_successful_decrypting_time_in_ms	2,374 (-6%)	4,637 (-5%)
note_history_trial_decrypting_time_in_ms	113 (+26%)	231 (+46%)
node_database_size_in_bytes	17,281,104	33,243,216 (+1%)
pxe_database_size_in_bytes	29,923	59,478

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	307	44,528	25,825
private-kernel-ordering	197	44,145	16,193
base-rollup	1,455	130,286	933
root-rollup	82.4 (-2%)	4,192	729
private-kernel-inner	437	71,331	25,825
public-kernel-private-input	249	34,687	27,457
public-kernel-non-first-iteration	247	34,729	27,457
merge-rollup	7.80	2,712	933

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	2 leaves	8 leaves	16 leaves	32 leaves	128 leaves	64 leaves	512 leaves	1024 leaves	2048 leaves	8192 leaves
batch_insert_into_append_only_tree_16_depth_ms	10.2	11.1 (+5%)	12.6 (-2%)	17.2 (+3%)	22.9 (+1%)	63.5 (-1%)	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.9	17.5	23.0	31.6	47.0	143	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.592	0.622 (+5%)	0.540 (-1%)	0.533 (+3%)	0.479	0.440	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	N/A	N/A	N/A	74.2	47.4	239 (-1%)	461	898	3,549
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	N/A	N/A	N/A	159	96.0	543	1,055	2,079	8,223
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.459	0.487	0.436	0.432	0.428	0.427
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	N/A	N/A	N/A	104 (+1%)	56.4	351 (+1%)	687	1,367 (+1%)	5,443 (+1%)
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	N/A	N/A	N/A	197	104	691	1,363	2,707	10,771
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.499 (+1%)	0.495	0.475	0.473	0.475 (+1%)	0.474 (+1%)
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	56.5	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	94.1	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	0.578	N/A	N/A	N/A	N/A	N/A	N/A	N/A

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	16,859	40,030

Transaction processing duration by data writes.

Metric	0 new commitments	1 new commitments
tx_pxe_processing_time_ms	553	1,311

Metric	0 public data writes	1 public data writes
tx_sequencer_processing_time_ms	0.524 (+8%)	595

[nventuro]

The cli CI job is failing due to an internal issue in our AWS ECR. I'm still looking into what's wrong with the boxes job.

[benesjan]

Just minor comments. Once we have the note id counter macro it will look sleek. Great job!

[LHerskind]

I think it might had been simpler or looking less "random" if using something like the selector computation to get the randomness 🤷

But otherwise looks good to me, trying to figure out why it fails the boxes. At some point the boxes was pulling from what was published on master, might be a leftover from that that have caused it 🤔

[LHerskind]

This is just the right amount of chaotic energy