feat: Aztec prelude

[vezenovm]

Summary

Resolves #2837

Adds an aztec prelude similar to the stdlib prelude in Noir or Rust. Now when contracts have an aztec dependency developers do not need to directly import the types specified in the prelude, they are just automatically imported by any contract using aztec-nr. A developer can still import the types if they like but I think it is nice that a developer no longer needs to import types such as AztecAddress, state vars, etc., and can instead just start working with the aztec dep directly.

Changes

The MacroProcessor in Noir operates over an untyped AST before name resolution or def collection, and a typed AST that is processed after name resolution and before type checking.

Attempting to make changes to the untyped AST before collection of definitions means we only have the AST of the root file in our crate and are missing the definitions in lower modules of the crate. This PR lets us add definitions to the definition collector for a crate as part of Noir compilation before name resolution.

This PR focuses only on adding an aztec prelude and is only adding to the collected imports. However, this process can be expanded to support injecting other objects into the definition collector. I wanted to start with a simple example and an aztec prelude seemed like a good starting point.

EDIT: I decided to add a prelude.nr into aztec-nr so it is not hidden from those working on aztec-nr. The macro now simply checks whether we have a certain dependency and returns an extra preludes we want to inject using Noir's already preexisting process for adding preludes. Due to this I had to keep the old strategy for imported use dep::aztec; into every contract for the aztec macros. This could perhaps be improved to use a different name so that we can just use prelude.nr exclusively, but I felt this would be better left for a separate PR. cc'ing @Maddiaa0

Another goal of this PR is to also provide a simple example of how to re-use Noir's parser to create the AST types we would like to insert instead of having a bunch of helpers such as call(..) like we currently do in aztec_macros/lib.rs. Hopefully switching to this strategy will also speed up macro development. @nventuro is building on this with his work to remove compute_note_hash_and_nullifier.

For now I just removed the imports from the noir-contracts but this could be extended to anywhere we see dep::aztec.

Additional Context

It would be nice to automatically include the aztec dependency for Aztec contracts, however, this would touch the noirc driver rather than simply the noirc frontend and would require a different process for enabling additional std dependencies.

[AztecBot]

Benchmark results

Metrics with a significant change:

• batch_insert_into_append_only_tree_16_depth_ms (8): 16.9 (+34%)
• batch_insert_into_append_only_tree_16_depth_hash_ms (8): 0.717 (+35%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 8619c084 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	64 txs
l1_rollup_calldata_size_in_bytes	5,700	18,884	36,452
l1_rollup_calldata_gas	66,132	238,976	469,676
l1_rollup_execution_gas	194,080	500,282	908,934
l2_block_processing_time_in_ms	1,198 (-3%)	4,520 (-2%)	9,015 (-1%)
note_successful_decrypting_time_in_ms	202 (-1%)	529 (-6%)	1,023 (-2%)
note_trial_decrypting_time_in_ms	9.91 (-34%)	59.2 (-37%)	61.7 (-5%)
l2_block_building_time_in_ms	16,142 (-1%)	64,122	128,546
l2_block_rollup_simulation_time_in_ms	12,245 (-1%)	48,725	97,683
l2_block_public_tx_process_time_in_ms	3,866 (-1%)	15,321	30,689 (+1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	14,136 (-5%)	27,643
note_history_successful_decrypting_time_in_ms	1,241 (-1%)	2,395 (-1%)
note_history_trial_decrypting_time_in_ms	95.4 (-2%)	135 (-13%)
node_database_size_in_bytes	18,796,624	35,475,536
pxe_database_size_in_bytes	29,923	59,478

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	249	44,736	28,001
private-kernel-ordering	178 (-1%)	52,625	14,627
base-rollup	1,297	177,932	933
root-rollup	71.3	4,192	825
private-kernel-inner	316	73,715	28,001
public-kernel-app-logic	193 (+1%)	32,254	25,379
merge-rollup	5.78 (-1%)	2,712	933

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	2 leaves	8 leaves	16 leaves	32 leaves	64 leaves	128 leaves	512 leaves	1024 leaves	2048 leaves	4096 leaves
batch_insert_into_append_only_tree_16_depth_ms	9.88 (-1%)	10.9 (-1%)	⚠️ 16.9 (+34%)	16.1 (-4%)	22.0 (-13%)	35.5	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.9	17.5	23.0	31.6	47.0	79.0	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.573	0.603	⚠️ 0.717 (+35%)	0.497 (-3%)	0.461 (-12%)	0.442	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	N/A	N/A	N/A	45.9	72.2 (-2%)	230 (-3%)	442 (-1%)	867 (-1%)	1,723 (-1%)
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	N/A	N/A	N/A	96.0	159	543	1,055	2,079	4,127
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.470	0.445 (-2%)	0.420 (-2%)	0.414 (-1%)	0.412 (-1%)	0.412
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	N/A	N/A	N/A	54.0 (-1%)	108 (-1%)	337 (-3%)	656 (-1%)	1,318	2,624
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	N/A	N/A	N/A	104	207	691	1,363	2,707	5,395
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.479	0.485 (-1%)	0.455 (-2%)	0.452 (-1%)	0.456	0.454
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	N/A	61.6	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	N/A	109	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	N/A	0.538	N/A	N/A	N/A	N/A	N/A	N/A

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts
tx_size_in_bytes	19,179

Transaction processing duration by data writes.

Metric	0 new note hashes	1 new note hashes
tx_pxe_processing_time_ms	2,601 (-1%)	1,364

Metric	0 public data writes	1 public data writes
tx_sequencer_processing_time_ms	0.0344 (+2%)	471

[Maddiaa0]

Over all an extremely nice example of this! If we have some stuff colliding with function names, i think it may be worth not including them in the prelude to prevent user frustration.

[rahul-kothari]

This is great work!
Would it be at all possible to include everything with a aztec::? e.g. aztec::get_portal_address()? This way we do not pollute keywords that devs might not know about?

curious to hear @iAmMichaelConnor opinion as well

[benesjan]

Current dependencies on/for this PR:

• feat: Aztec prelude #4496 👈
• master

This stack of pull requests is managed by Graphite.

Boo!

[AztecBot]

Docs Preview

Hey there! 👋 You can check your preview at https://65e5e9bc5761583e3cf60081--aztec-docs-dev.netlify.app

[TomAFrench]

I'm concerned that this adds a little too much "magic" to how we use these types as well as adding a lot of potential for conflicts in future. If every macro processor can add their own preludes then it's almost guaranteed that they're going to clash, especially with types as general as Map.

In general, I'm not a fan of importing a large number of types/functions from a library just because we're depending on it. Instead this should be opt in, if we look at preludes from Rust libraries for instance we'd need an use aztec::prelude::* import to make use of these types.

It's important to note that while they're core types to the aztec contract writing experience, from Noir's perspective aztec-nr is fundamentally just another library.

[TomAFrench]

I would prefer that for now we just keep the prelude file which would simplify users performing the imports themselves (as they don't need to remember exactly where in aztec-nr their import is defined).

Alternatively, in contracts decorated with #[aztec] we could inject an import of the prelude (although this may be confusing on why the prelude exists in the contract but not outside of it.

[benesjan]

If every macro processor can add their own preludes then it's almost guaranteed that they're going to clash, especially with types as general as Map.

@TomAFrench I don't know what is the best solution implementation-wise here but not including the PR and putting the burden of importing these basic types on aztec contract devs doesn't sound good to me. But if you feel like at this point this would cause more issues then benefits I am ok with not merging it now. But I think once things are more stable we should have something like this (at least for types like AztecAddress).

if we look at preludes from Rust libraries for instance we'd need an use aztec::prelude::* import to make use of these types.

@TomAFrench In Solidity you can just use types like address directly and I feel like solidity devs are the core audience here. Looking at a contract as a non-rust dev and seeing that I have to import some prelude thing I would have no idea what's going on. But if you feel that some of the types there are too generic I would not mind removing them.

It's important to note that while they're core types to the aztec contract writing experience, from Noir's perspective aztec-nr is fundamentally just another library.

@TomAFrench Why does this matter when the aztec macros are only used for aztec contracts?

[TomAFrench]

In Solidity you can just use types like address directly and I feel like solidity devs are the core audience here. Looking at a contract as a non-rust dev and seeing that I have to import some prelude thing I would have no idea what's going on. But if you feel that some of the types there are too generic I would not mind removing them.

I don't think that Solidity is the right comparison here as the language and the smart contract development framework are one and the same - there's no non smart contract solidity development. A better comparison in my mind would be Arbitrum Stylus or Solana as they're both using a general purpose langauge with a smart contract framework built on top, in both of these cases it's necessary to import from a library to get smart contract types, etc.

Why does this matter when the aztec macros are only used for aztec contracts?

Because the ultimate aim is to have the macro logic expressed in Noir source code such that the aztec macros can be included in the aztec-nr crate. At this point then any crate can implement similar logic to aztec-nr so if aztec-nr can arbitrarily add imports then so can every other crate you depend on.

[vezenovm]

especially with types as general as Map.

We can remove these general names if we would like or just use an alias such as AztecMap.

In general, I'm not a fan of importing a large number of types/functions from a library just because we're depending on it. Instead this should be opt in, if we look at preludes from Rust libraries for instance we'd need an use aztec::prelude::* import to make use of these types.

It's important to note that while they're core types to the aztec contract writing experience, from Noir's perspective aztec-nr is fundamentally just another library.

I agree! However, these statements can be applied to the entirety of aztec_macros not just the prelude itself. The macros package already injects imports and code based upon those imports because we are dependent on aztec-nr. This is what the initial macro #[aztec(private)] does. The prelude isn't adding new functionality in this sense, but rather extending what already exists.

I'm ok with making this opt-in but its feels like it would actually add confusion for users with some of the new macros the aztec team is working to add. For example here #4610 the NoteInterface trait and note::utils::compute_note_hash_and_nullifier essentially became required names. I find it more confusing that someone would have to import a function that is not written explicitly anywhere in the contract.

in both of these cases it's necessary to import from a library to get smart contract types, etc.

We can improve this UX though. We should maintain the boundary of library and language, but we don't yet have full macros on the Noir side that can enable this.

Because the ultimate aim is to have the macro logic expressed in Noir source code such that the aztec macros can be included in the aztec-nr crate. At this point then any crate can implement similar logic to aztec-nr so if aztec-nr can arbitrarily add imports then so can every other crate you depend on.

Yes absolutely. But we are not at this point yet, and we do not have any timeline for it right now. And with new macros potentially using reserved names, aztec-nr should have a prelude as mentioned above. This feels like a more general question of whether we should commit fully to Noir macros and pause development on the aztec_macros package.

[vezenovm]

@TomAFrench Most of these comments feel like reasons as to why we should move to get rid of aztec_macros itself, but not whether a prelude makes sense for aztec-nr.

We should set aside a discussion next week on the future of aztec_macros as well as get some context on future macros the aztec team plans to add. But none of this feels blocking for this PR.

[vezenovm]

This was succeeded by #4929 so i am closing.