Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: public data tree as indexed tree #3566

Merged
merged 54 commits into from
Dec 14, 2023
Merged
Changes from 1 commit
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
880b702
wip abstracting leaf and preimage in indexed
sirasistant Dec 1, 2023
ed471d4
feat: experiment with generic indexed trees
sirasistant Dec 1, 2023
38fcf05
chore: add test dep to devDependencies
sirasistant Dec 1, 2023
bc4915b
chore: remove old LeafData struct
sirasistant Dec 1, 2023
47a3aee
fix: getLatestLeafPreimage can return undefined
sirasistant Dec 1, 2023
064d7a8
feat: remove local leaf cache in indexed tree
sirasistant Dec 1, 2023
d458756
feat: optimize findLeafIndex for indexed tree
sirasistant Dec 1, 2023
ae6b6a0
docs: added jsdoc
sirasistant Dec 4, 2023
47bb7f0
refactor: low leaf getter returns option
sirasistant Dec 4, 2023
6d7fc7d
Merge branch 'master' into arv/public_data_tree_in
sirasistant Dec 4, 2023
7250939
fix: make sure low leaf in range
sirasistant Dec 4, 2023
240e8b5
fix: improve performance of getLeafIndex
sirasistant Dec 4, 2023
bcc8ae0
chore: cleanups
sirasistant Dec 4, 2023
1b7c9f8
style: fix formatting
sirasistant Dec 4, 2023
3848116
Merge branch 'master' into arv/public_data_tree_indexed
sirasistant Dec 4, 2023
08791c5
chore: reset file to master
sirasistant Dec 4, 2023
989ad09
Merge branch 'master' into arv/public_data_tree_indexed
sirasistant Dec 4, 2023
678bc75
feat: public data tree switch
sirasistant Dec 4, 2023
a7c2148
chore: update constants gen
sirasistant Dec 4, 2023
fd4ffde
Merge branch 'master' into arv/upsert_indexed_tree
sirasistant Dec 4, 2023
1d85666
feat: more work towards switching data tree
sirasistant Dec 5, 2023
edb4548
fix: addressed some PR comments
sirasistant Dec 5, 2023
e731f97
fix: avoid findLeafIndex in sparse trees
sirasistant Dec 5, 2023
037a479
refactor: avoid createTree helper
sirasistant Dec 5, 2023
a217fe2
fix: formatting
sirasistant Dec 5, 2023
3d6cfb4
feat: more work towards indexed pdt
sirasistant Dec 5, 2023
2a77b8b
Merge branch 'master' into arv/public_data_tree_indexed
sirasistant Dec 5, 2023
8be1d4a
Merge branch 'arv/public_data_tree_indexed' into arv/upsert_indexed_tree
sirasistant Dec 5, 2023
02d6e6b
fix: bugfixes in upsert and public data reads
sirasistant Dec 5, 2023
3337297
test: skipping a sol test for now
sirasistant Dec 5, 2023
3a95f1b
fix: formatting
sirasistant Dec 5, 2023
b4998e2
fix: getStorageAt returns zero if not inserted
sirasistant Dec 5, 2023
7f07897
fix: remove redundant writes
sirasistant Dec 5, 2023
32be31c
fix: sync properly the public data tree
sirasistant Dec 5, 2023
a6d10c7
fix: sync public data tree updates
sirasistant Dec 6, 2023
4897b98
feat: add function for verifying reads
sirasistant Dec 6, 2023
c186490
test: some unit test fixes
sirasistant Dec 12, 2023
bc021b9
test: fix block builder unit test
sirasistant Dec 12, 2023
4e603fc
feat: fix inclusion proofs
sirasistant Dec 12, 2023
65a4762
Merge branch 'master' into arv/upsert_indexed_tree
sirasistant Dec 12, 2023
dbd2c1e
fix: noir tests for public data tree
sirasistant Dec 13, 2023
af75d19
Merge branch 'master' into arv/upsert_indexed_tree
sirasistant Dec 13, 2023
1a8bc3d
test: add unit test for upsertion in indexed tree
sirasistant Dec 13, 2023
01e8b35
chore: cleanups and refactors
sirasistant Dec 13, 2023
0c416aa
chore: update snaps and doc
sirasistant Dec 13, 2023
f91c762
chore: docs
sirasistant Dec 13, 2023
1342ca5
test: re enable solidity test
sirasistant Dec 13, 2023
f191c2e
test: refactor noir tests
sirasistant Dec 13, 2023
98fa494
Merge branch 'master' into arv/upsert_indexed_tree
sirasistant Dec 13, 2023
4cf3644
Merge branch 'master' into arv/upsert_indexed_tree
sirasistant Dec 13, 2023
cc48e39
Merge branch 'arv/upsert_indexed_tree' of github.com:AztecProtocol/az…
sirasistant Dec 13, 2023
83e6adb
Merge branch 'master' into arv/upsert_indexed_tree
sirasistant Dec 13, 2023
40ba830
Merge branch 'master' into arv/upsert_indexed_tree
sirasistant Dec 14, 2023
14705ec
Merge branch 'master' into arv/upsert_indexed_tree
sirasistant Dec 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' into arv/public_data_tree_in
sirasistant committed Dec 4, 2023
commit 6d7fc7d3a394754778b14b1e379d2c018567d2cd
19 changes: 6 additions & 13 deletions yarn-project/aztec-node/src/aztec-node/server.ts
Original file line number Diff line number Diff line change
@@ -8,7 +8,6 @@ import {
L1_TO_L2_MSG_TREE_HEIGHT,
NOTE_HASH_TREE_HEIGHT,
NULLIFIER_TREE_HEIGHT,
NullifierLeaf,
NullifierLeafPreimage,
PUBLIC_DATA_TREE_HEIGHT,
} from '@aztec/circuits.js';
@@ -431,11 +430,8 @@ export class AztecNodeService implements AztecNode {
return undefined;
}

const leafDataPromise = committedDb.getLeafPreimage<NullifierLeaf, NullifierLeafPreimage>(
MerkleTreeId.NULLIFIER_TREE,
index,
);
const siblingPathPromise = committedDb.getSiblingPath<typeof NULLIFIER_TREE_HEIGHT>(
const leafDataPromise = db.getLeafPreimage(MerkleTreeId.NULLIFIER_TREE, index);
const siblingPathPromise = db.getSiblingPath<typeof NULLIFIER_TREE_HEIGHT>(
MerkleTreeId.NULLIFIER_TREE,
BigInt(index),
);
@@ -446,7 +442,7 @@ export class AztecNodeService implements AztecNode {
return undefined;
}

return new NullifierMembershipWitness(BigInt(index), leafData, siblingPath);
return new NullifierMembershipWitness(BigInt(index), leafData as NullifierLeafPreimage, siblingPath);
}

/**
@@ -467,7 +463,7 @@ export class AztecNodeService implements AztecNode {
blockNumber: number | 'latest',
nullifier: Fr,
): Promise<NullifierMembershipWitness | undefined> {
const committedDb = await this.#getWorldState();
const committedDb = await this.#getWorldState(blockNumber);
const findResult = await committedDb.getPreviousValueIndex(MerkleTreeId.NULLIFIER_TREE, nullifier.toBigInt());
if (!findResult) {
return undefined;
@@ -476,16 +472,13 @@ export class AztecNodeService implements AztecNode {
if (alreadyPresent) {
this.log.warn(`Nullifier ${nullifier.toBigInt()} already exists in the tree`);
}
const preimageData = (await committedDb.getLeafPreimage<NullifierLeaf, NullifierLeafPreimage>(
MerkleTreeId.NULLIFIER_TREE,
index,
))!;
const preimageData = (await committedDb.getLeafPreimage(MerkleTreeId.NULLIFIER_TREE, index))!;

const siblingPath = await committedDb.getSiblingPath<typeof NULLIFIER_TREE_HEIGHT>(
MerkleTreeId.NULLIFIER_TREE,
BigInt(index),
);
return new NullifierMembershipWitness(BigInt(index), preimageData, siblingPath);
return new NullifierMembershipWitness(BigInt(index), preimageData as NullifierLeafPreimage, siblingPath);
}

/**
2 changes: 1 addition & 1 deletion yarn-project/circuits.js/src/structs/rollup/base_rollup.ts
Original file line number Diff line number Diff line change
@@ -28,7 +28,7 @@ import { AppendOnlyTreeSnapshot } from './append_only_tree_snapshot.js';
* Class containing the data of a preimage of a single leaf in the nullifier tree.
* Note: It's called preimage because this data gets hashed before being inserted as a node into the `IndexedTree`.
*/
export class NullifierLeafPreimage implements IndexedTreeLeafPreimage<NullifierLeaf> {
export class NullifierLeafPreimage implements IndexedTreeLeafPreimage {
constructor(
/**
* Leaf value inside the indexed tree's linked list.
4 changes: 2 additions & 2 deletions yarn-project/foundation/src/trees/index.ts
Original file line number Diff line number Diff line change
@@ -19,7 +19,7 @@ export interface IndexedTreeLeaf {
/**
* Preimage of an indexed merkle tree leaf.
*/
export interface IndexedTreeLeafPreimage<Leaf extends IndexedTreeLeaf> {
export interface IndexedTreeLeafPreimage {
/**
* Returns key of the leaf corresponding to this preimage.
*/
@@ -36,7 +36,7 @@ export interface IndexedTreeLeafPreimage<Leaf extends IndexedTreeLeaf> {
/**
* Returns the preimage as a leaf.
*/
asLeaf(): Leaf;
asLeaf(): IndexedTreeLeaf;
/**
* Serializes the preimage into a buffer.
*/
2 changes: 1 addition & 1 deletion yarn-project/merkle-tree/src/index.ts
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@ export * from './interfaces/merkle_tree.js';
export * from './interfaces/update_only_tree.js';
export * from './pedersen.js';
export * from './sparse_tree/sparse_tree.js';
export { LowLeafWitnessData, StandardIndexedTree } from './standard_indexed_tree/standard_indexed_tree.js';
export { StandardIndexedTree } from './standard_indexed_tree/standard_indexed_tree.js';
export * from './standard_tree/standard_tree.js';
export { INITIAL_LEAF } from './tree_base.js';
export { newTree, builder as treeBuilder } from './new_tree.js';
29 changes: 8 additions & 21 deletions yarn-project/merkle-tree/src/interfaces/indexed_tree.ts
Original file line number Diff line number Diff line change
@@ -1,20 +1,16 @@
import { IndexedTreeLeaf, IndexedTreeLeafPreimage } from '@aztec/foundation/trees';
import { IndexedTreeLeafPreimage } from '@aztec/foundation/trees';
import { SiblingPath } from '@aztec/types';

import { AppendOnlyTree } from './append_only_tree.js';

/**
* All of the data to be return during batch insertion.
*/
export interface LowLeafWitnessData<
N extends number,
Leaf extends IndexedTreeLeaf,
Preimage extends IndexedTreeLeafPreimage<Leaf>,
> {
export interface LowLeafWitnessData<N extends number> {
/**
* Preimage of the low nullifier that proves non membership.
*/
leafData: Preimage;
leafData: IndexedTreeLeafPreimage;
/**
* Sibling path to prove membership of low nullifier.
*/
@@ -28,16 +24,11 @@ export interface LowLeafWitnessData<
/**
* The result of a batch insertion in an indexed merkle tree.
*/
export interface BatchInsertionResult<
TreeHeight extends number,
SubtreeSiblingPathHeight extends number,
Leaf extends IndexedTreeLeaf,
Preimage extends IndexedTreeLeafPreimage<Leaf>,
> {
export interface BatchInsertionResult<TreeHeight extends number, SubtreeSiblingPathHeight extends number> {
/**
* Data for the leaves to be updated when inserting the new ones.
*/
lowLeavesWitnessData?: LowLeafWitnessData<TreeHeight, Leaf, Preimage>[];
lowLeavesWitnessData?: LowLeafWitnessData<TreeHeight>[];
/**
* Sibling path "pointing to" where the new subtree should be inserted into the tree.
*/
@@ -55,8 +46,7 @@ export interface BatchInsertionResult<
/**
* Indexed merkle tree.
*/
export interface IndexedTree<Leaf extends IndexedTreeLeaf, Preimage extends IndexedTreeLeafPreimage<Leaf>>
extends AppendOnlyTree {
export interface IndexedTree extends AppendOnlyTree {
/**
* Finds the index of the largest leaf whose value is less than or equal to the provided value.
* @param newValue - The new value to be inserted into the tree.
@@ -86,10 +76,7 @@ export interface IndexedTree<Leaf extends IndexedTreeLeaf, Preimage extends Inde
* @param includeUncommitted - If true, the uncommitted changes are included in the search.
* @returns A copy of the leaf preimage at the given index or undefined if the leaf was not found.
*/
getLatestLeafPreimageCopy(
index: bigint,
includeUncommitted: boolean,
): Promise<IndexedTreeLeafPreimage<Leaf> | undefined>;
getLatestLeafPreimageCopy(index: bigint, includeUncommitted: boolean): Promise<IndexedTreeLeafPreimage | undefined>;

/**
* Batch insert multiple leaves into the tree.
@@ -101,5 +88,5 @@ export interface IndexedTree<Leaf extends IndexedTreeLeaf, Preimage extends Inde
leaves: Buffer[],
subtreeHeight: SubtreeHeight,
includeUncommitted: boolean,
): Promise<BatchInsertionResult<TreeHeight, SubtreeSiblingPathHeight, Leaf, Preimage>>;
): Promise<BatchInsertionResult<TreeHeight, SubtreeSiblingPathHeight>>;
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { TreeSnapshotBuilder } from '../snapshots/snapshot_builder.js';
import { MerkleTree } from './merkle_tree.js';

/**
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import levelup, { LevelUp } from 'levelup';

import { Pedersen, StandardTree, newTree } from '../index.js';
import { Pedersen, StandardTree, newTree, treeBuilder } from '../index.js';
import { createMemDown } from '../test/utils/create_mem_down.js';
import { AppendOnlySnapshotBuilder } from './append_only_snapshot.js';
import { describeSnapshotBuilderTestSuite } from './snapshot_builder_test_suite.js';
@@ -13,7 +13,7 @@ describe('AppendOnlySnapshot', () => {
beforeEach(async () => {
db = levelup(createMemDown());
const hasher = new Pedersen();
tree = await newTree(StandardTree, db, hasher, 'test', 4);
tree = await newTree(treeBuilder(StandardTree), db, hasher, 'test', 4);
snapshotBuilder = new AppendOnlySnapshotBuilder(db, tree, hasher);
});

4 changes: 2 additions & 2 deletions yarn-project/merkle-tree/src/snapshots/base_full_snapshot.ts
Original file line number Diff line number Diff line change
@@ -65,7 +65,7 @@ export abstract class BaseFullTreeSnapshotBuilder<T extends TreeBase, S extends
if (level + 1 > depth) {
// short circuit if we've reached the leaf level
// otherwise getNode might throw if we ask for the children of a leaf
this.handleLeaf(i, node, batch);
await this.handleLeaf(i, node, batch);
continue;
}

@@ -98,7 +98,7 @@ export abstract class BaseFullTreeSnapshotBuilder<T extends TreeBase, S extends
}

protected handleLeaf(_index: bigint, _node: Buffer, _batch: LevelUpChain) {
return;
return Promise.resolve();
}

async getSnapshot(version: number): Promise<S> {
4 changes: 2 additions & 2 deletions yarn-project/merkle-tree/src/snapshots/full_snapshot.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import levelup, { LevelUp } from 'levelup';

import { Pedersen, StandardTree, newTree } from '../index.js';
import { Pedersen, StandardTree, newTree, treeBuilder } from '../index.js';
import { createMemDown } from '../test/utils/create_mem_down.js';
import { FullTreeSnapshotBuilder } from './full_snapshot.js';
import { describeSnapshotBuilderTestSuite } from './snapshot_builder_test_suite.js';
@@ -12,7 +12,7 @@ describe('FullSnapshotBuilder', () => {

beforeEach(async () => {
db = levelup(createMemDown());
tree = await newTree(StandardTree, db, new Pedersen(), 'test', 4);
tree = await newTree(treeBuilder(StandardTree), db, new Pedersen(), 'test', 4);
snapshotBuilder = new FullTreeSnapshotBuilder(db, tree);
});

Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
import { Fr, NullifierLeaf, NullifierLeafPreimage } from '@aztec/circuits.js';
import { Hasher } from '@aztec/types';

import levelup, { LevelUp } from 'levelup';

import { Pedersen, newTree } from '../index.js';
@@ -13,15 +16,23 @@ describe('IndexedTreeSnapshotBuilder', () => {

beforeEach(async () => {
db = levelup(createMemDown());
tree = await newTree(StandardIndexedTreeWithAppend, db, new Pedersen(), 'test', 4);
snapshotBuilder = new IndexedTreeSnapshotBuilder(db, tree);
tree = await newTree(
(db: levelup.LevelUp, hasher: Hasher, name: string, depth: number, size: bigint) => {
return new StandardIndexedTreeWithAppend(db, hasher, name, depth, size, NullifierLeafPreimage, NullifierLeaf);
},
db,
new Pedersen(),
'test',
4,
);
snapshotBuilder = new IndexedTreeSnapshotBuilder(db, tree, NullifierLeafPreimage);
});

describeSnapshotBuilderTestSuite(
() => tree,
() => snapshotBuilder,
async () => {
const newLeaves = Array.from({ length: 2 }).map(() => Buffer.from(Math.random().toString()));
const newLeaves = Array.from({ length: 2 }).map(() => new NullifierLeaf(Fr.random()).toBuffer());
await tree.appendLeaves(newLeaves);
},
);
@@ -31,50 +42,50 @@ describe('IndexedTreeSnapshotBuilder', () => {
await tree.appendLeaves([Buffer.from('a'), Buffer.from('b'), Buffer.from('c')]);
await tree.commit();
const expectedLeavesAtBlock1 = await Promise.all([
tree.getLatestLeafDataCopy(0, false),
tree.getLatestLeafDataCopy(1, false),
tree.getLatestLeafDataCopy(2, false),
tree.getLatestLeafPreimageCopy(0n, false),
tree.getLatestLeafPreimageCopy(1n, false),
tree.getLatestLeafPreimageCopy(2n, false),
// id'expect these to be undefined, but leaf 3 isn't?
// must be some indexed-tree quirk I don't quite understand yet
tree.getLatestLeafDataCopy(3, false),
tree.getLatestLeafDataCopy(4, false),
tree.getLatestLeafDataCopy(5, false),
tree.getLatestLeafPreimageCopy(3n, false),
tree.getLatestLeafPreimageCopy(4n, false),
tree.getLatestLeafPreimageCopy(5n, false),
]);

await snapshotBuilder.snapshot(1);

await tree.appendLeaves([Buffer.from('d'), Buffer.from('e'), Buffer.from('f')]);
await tree.commit();
const expectedLeavesAtBlock2 = await Promise.all([
tree.getLatestLeafDataCopy(0, false),
tree.getLatestLeafDataCopy(1, false),
tree.getLatestLeafDataCopy(2, false),
tree.getLatestLeafDataCopy(3, false),
tree.getLatestLeafDataCopy(4, false),
tree.getLatestLeafDataCopy(5, false),
tree.getLatestLeafPreimageCopy(0n, false),
tree.getLatestLeafPreimageCopy(1n, false),
tree.getLatestLeafPreimageCopy(2n, false),
tree.getLatestLeafPreimageCopy(3n, false),
tree.getLatestLeafPreimageCopy(4n, false),
tree.getLatestLeafPreimageCopy(5n, false),
]);

await snapshotBuilder.snapshot(2);

const snapshot1 = await snapshotBuilder.getSnapshot(1);
const actualLeavesAtBlock1 = await Promise.all([
snapshot1.getLatestLeafDataCopy(0n),
snapshot1.getLatestLeafDataCopy(1n),
snapshot1.getLatestLeafDataCopy(2n),
snapshot1.getLatestLeafDataCopy(3n),
snapshot1.getLatestLeafDataCopy(4n),
snapshot1.getLatestLeafDataCopy(5n),
snapshot1.getLatestLeafPreimageCopy(0n),
snapshot1.getLatestLeafPreimageCopy(1n),
snapshot1.getLatestLeafPreimageCopy(2n),
snapshot1.getLatestLeafPreimageCopy(3n),
snapshot1.getLatestLeafPreimageCopy(4n),
snapshot1.getLatestLeafPreimageCopy(5n),
]);
expect(actualLeavesAtBlock1).toEqual(expectedLeavesAtBlock1);

const snapshot2 = await snapshotBuilder.getSnapshot(2);
const actualLeavesAtBlock2 = await Promise.all([
snapshot2.getLatestLeafDataCopy(0n),
snapshot2.getLatestLeafDataCopy(1n),
snapshot2.getLatestLeafDataCopy(2n),
snapshot2.getLatestLeafDataCopy(3n),
snapshot2.getLatestLeafDataCopy(4n),
snapshot2.getLatestLeafDataCopy(5n),
snapshot2.getLatestLeafPreimageCopy(0n),
snapshot2.getLatestLeafPreimageCopy(1n),
snapshot2.getLatestLeafPreimageCopy(2n),
snapshot2.getLatestLeafPreimageCopy(3n),
snapshot2.getLatestLeafPreimageCopy(4n),
snapshot2.getLatestLeafPreimageCopy(5n),
]);
expect(actualLeavesAtBlock2).toEqual(expectedLeavesAtBlock2);
});
@@ -85,7 +96,7 @@ describe('IndexedTreeSnapshotBuilder', () => {
await tree.appendLeaves([Buffer.from('a'), Buffer.from('f'), Buffer.from('d')]);
await tree.commit();
const snapshot = await snapshotBuilder.snapshot(1);
const historicalPrevValue = tree.findIndexOfPreviousValue(2n, false);
const historicalPrevValue = await tree.findIndexOfPreviousKey(2n, false);

await tree.appendLeaves([Buffer.from('c'), Buffer.from('b'), Buffer.from('e')]);
await tree.commit();
Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.