feat: persistent archiver store #3410

alexghr · 2023-11-23T11:59:35Z

This PR adds a new implementation for an archiver store that's backed by LMDB. See #3361 for details.

socket-security · 2023-11-23T12:17:48Z

New dependencies detected. Learn more about Socket for GitHub ↗︎

Packages	Version	New capabilities	Transitives	Size	Publisher
lmdb	2.9.1	eval, filesystem, environment	`+54`	41.4 MB	kriszyp

alexghr · 2023-11-23T14:41:47Z

yarn-project/end-to-end/src/sample-dapp/contracts.mjs

 import { TokenContractArtifact } from '@aztec/noir-contracts/artifacts';

 import { readFileSync } from 'fs';

 // docs:start:get-tokens
 export async function getToken(client) {
  const addresses = JSON.parse(readFileSync('addresses.json'));
-  return Contract.at(addresses.token, TokenContractArtifact, client);
+  return Contract.at(AztecAddress.fromString(addresses.token), TokenContractArtifact, client);


Turns out we were passing a string when Contract.at wants an instance of AztecAddress and the only reason this worked was because the old MemoryArchiverStore was using address.toString() to find the contract. The new store actually really needs an address instance so this was crashing.

Might be worth going through all of the JS tests files and adding jsdoc typings so we at least spot issues in the editor.

alexghr · 2023-11-23T14:43:57Z

yarn-project/archiver/src/archiver/archiver.ts

+    const archiverStore = rootDB
+      ? new LMDBArchiverStore(rootDB, config.maxLogs ?? 1000)
+      : new MemoryArchiverStore(config.maxLogs ?? 1000);


I've left the old MemoryArchiverStore in as a reference for the code review and in case we might need it for other tests but otherwise all code paths should pass a rootDB now and use a temporary LMDB instance if persistance isn't needed.

I'm wondering if this function should just accept the ArchiverStore interface and the concrete implementation is created at node setup. Otherwise, if I wanted to add a different DB implementation I would need to pollute the arguments.

AztecBot · 2023-11-24T09:01:01Z

Benchmark results

Metrics with a significant change:

node_database_size_in_bytes (5): 3,321,047 (+104%)
node_database_size_in_bytes (10): 3,661,042 (+233%)
note_history_trial_decrypting_time_in_ms (10): 231 (+46%)
note_trial_decrypting_time_in_ms (8): 38.3 (+310%)
note_trial_decrypting_time_in_ms (128): 199 (+40%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit c4e4745d and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,864	867,704	3,448,052
l1_rollup_execution_gas	841,951	3,594,812	22,203,421
l2_block_processing_time_in_ms	2,043 (+1%)	7,671 (+1%)	31,225 (+5%)
note_successful_decrypting_time_in_ms	330 (+7%)	973 (+8%)	3,676 (+13%)
note_trial_decrypting_time_in_ms	⚠️ 38.3 (+310%)	99.7 (-9%)	⚠️ 199 (+40%)
l2_block_building_time_in_ms	25,398 (+1%)	100,849	403,618 (+1%)
l2_block_rollup_simulation_time_in_ms	17,830 (+1%)	70,780	282,458 (+1%)
l2_block_public_tx_process_time_in_ms	7,520	29,922	120,604

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	22,661 (+2%)	44,006 (+2%)
note_history_successful_decrypting_time_in_ms	2,308 (+8%)	4,502 (+8%)
note_history_trial_decrypting_time_in_ms	160 (+28%)	⚠️ 231 (+46%)
node_database_size_in_bytes	⚠️ 3,321,047 (+104%)	⚠️ 3,661,042 (+233%)
pxe_database_size_in_bytes	29,748	59,307

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	399	43,077	20,441
private-kernel-ordering	126	25,833	9,689
base-rollup	3,183	659,500	873
root-rollup	97.7 (+5%)	4,072	1,097
private-kernel-inner	825	64,484	20,441
public-kernel-private-input	417	25,203	20,441
public-kernel-non-first-iteration	411	25,245	20,441
merge-rollup	11.1 (+2%)	2,592	873

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	10,323	29,083

alexghr · 2023-11-24T15:50:17Z

~~discovered an issue restarting a node with existing data. Will reopen the PR once I push a fix.~~

Tracked down issue to how the trees restored themselves from the LevelDB instance. Fix in these two commits https://github.com/AztecProtocol/aztec-packages/pull/3410/files/8fc557ed4966a16d46672c1b4984e1763ed8d6bb..a4235f93bc8c31fdd198f7b8bbb7857d28e3948e

PhilWindle · 2023-11-24T15:19:52Z

yarn-project/archiver/src/archiver/archiver.ts

+    const archiverStore = rootDB
+      ? new LMDBArchiverStore(rootDB, config.maxLogs ?? 1000)
+      : new MemoryArchiverStore(config.maxLogs ?? 1000);


I'm wondering if this function should just accept the ArchiverStore interface and the concrete implementation is created at node setup. Otherwise, if I wanted to add a different DB implementation I would need to pollute the arguments.

PhilWindle · 2023-11-24T15:21:03Z

yarn-project/archiver/src/archiver/archiver_store_test_suite.ts

+ * @param testName - The name of the test suite.
+ * @param getStore - Returns an instance of a store that's already been initialized.
+ */
+export function describeArchiverDataStore(testName: string, getStore: () => ArchiverDataStore) {


This is great!

…nt-archiver-store

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.16.1</summary> ## [0.16.1](aztec-packages-v0.16.0...aztec-packages-v0.16.1) (2023-11-28) ### Features * Added poseidon2 hash function to barretenberg/crypto ([#3118](#3118)) ([d47782b](d47782b)) * Aztec CI files in Noir ([#3430](#3430)) ([1621f3a](1621f3a)) * Persistent archiver store ([#3410](#3410)) ([4735bde](4735bde)), closes [#3361](#3361) ### Bug Fixes * **ci:** Don't leave DRY_DEPLOY unset ([#3449](#3449)) ([454e316](454e316)) * **ci:** Publishing dockerhub manifests ([#3451](#3451)) ([a59e7f0](a59e7f0)) * Hotfix noir sync ([#3436](#3436)) ([c4e4745](c4e4745)) ### Miscellaneous * **docs:** Core concepts page in getting-started ([#3401](#3401)) ([1a62f73](1a62f73)) * Point acir tests at noir master branch ([#3440](#3440)) ([106e690](106e690)) ### Documentation * Further updates to the gas and fees whitepaper ([#3448](#3448)) ([4152ba6](4152ba6)) * Updates to gas and fees yellow paper ([#3438](#3438)) ([5f0e1ca](5f0e1ca)) </details> <details><summary>barretenberg.js: 0.16.1</summary> ## [0.16.1](barretenberg.js-v0.16.0...barretenberg.js-v0.16.1) (2023-11-28) ### Miscellaneous * **barretenberg.js:** Synchronize aztec-packages versions </details> <details><summary>barretenberg: 0.16.1</summary> ## [0.16.1](barretenberg-v0.16.0...barretenberg-v0.16.1) (2023-11-28) ### Features * Added poseidon2 hash function to barretenberg/crypto ([#3118](#3118)) ([d47782b](d47782b)) ### Miscellaneous * Point acir tests at noir master branch ([#3440](#3440)) ([106e690](106e690)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

This PR adds a new implementation for an archiver store that's backed by LMDB. See #3361 for details. Fix #3361

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.16.1</summary> ## [0.16.1](AztecProtocol/aztec-packages@aztec-packages-v0.16.0...aztec-packages-v0.16.1) (2023-11-28) ### Features * Added poseidon2 hash function to barretenberg/crypto ([#3118](AztecProtocol/aztec-packages#3118)) ([d47782b](AztecProtocol/aztec-packages@d47782b)) * Aztec CI files in Noir ([#3430](AztecProtocol/aztec-packages#3430)) ([1621f3a](AztecProtocol/aztec-packages@1621f3a)) * Persistent archiver store ([#3410](AztecProtocol/aztec-packages#3410)) ([4735bde](AztecProtocol/aztec-packages@4735bde)), closes [#3361](AztecProtocol/aztec-packages#3361) ### Bug Fixes * **ci:** Don't leave DRY_DEPLOY unset ([#3449](AztecProtocol/aztec-packages#3449)) ([454e316](AztecProtocol/aztec-packages@454e316)) * **ci:** Publishing dockerhub manifests ([#3451](AztecProtocol/aztec-packages#3451)) ([a59e7f0](AztecProtocol/aztec-packages@a59e7f0)) * Hotfix noir sync ([#3436](AztecProtocol/aztec-packages#3436)) ([c4e4745](AztecProtocol/aztec-packages@c4e4745)) ### Miscellaneous * **docs:** Core concepts page in getting-started ([#3401](AztecProtocol/aztec-packages#3401)) ([1a62f73](AztecProtocol/aztec-packages@1a62f73)) * Point acir tests at noir master branch ([#3440](AztecProtocol/aztec-packages#3440)) ([106e690](AztecProtocol/aztec-packages@106e690)) ### Documentation * Further updates to the gas and fees whitepaper ([#3448](AztecProtocol/aztec-packages#3448)) ([4152ba6](AztecProtocol/aztec-packages@4152ba6)) * Updates to gas and fees yellow paper ([#3438](AztecProtocol/aztec-packages#3438)) ([5f0e1ca](AztecProtocol/aztec-packages@5f0e1ca)) </details> <details><summary>barretenberg.js: 0.16.1</summary> ## [0.16.1](AztecProtocol/aztec-packages@barretenberg.js-v0.16.0...barretenberg.js-v0.16.1) (2023-11-28) ### Miscellaneous * **barretenberg.js:** Synchronize aztec-packages versions </details> <details><summary>barretenberg: 0.16.1</summary> ## [0.16.1](AztecProtocol/aztec-packages@barretenberg-v0.16.0...barretenberg-v0.16.1) (2023-11-28) ### Features * Added poseidon2 hash function to barretenberg/crypto ([#3118](AztecProtocol/aztec-packages#3118)) ([d47782b](AztecProtocol/aztec-packages@d47782b)) ### Miscellaneous * Point acir tests at noir master branch ([#3440](AztecProtocol/aztec-packages#3440)) ([106e690](AztecProtocol/aztec-packages@106e690)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

alexghr added 7 commits November 23, 2023 10:11

feat: add jest helpers to @aztec/types

5ba9aff

refactor: split MemoryArchiverStore to own file

f3869b6

refactor: addLogs per block

de93bfc

test: add comprehensive archiver_store test suite

051549d

feat: add archiver store backed by LMDB

0b378e4

fix: await contracts added to the archiver store

bdf9c5c

feat: use persistent archiver store

1b82206

alexghr and others added 4 commits November 23, 2023 13:09

Merge branch 'master' into alexg/feat/persistent-archiver-store

d21cb49

fix: L1ToL2Message.random takes optional entryKey

02f924b

fix: typing error in ts-jest

d84ab77

fix: parse aztec address from string

fd95142

alexghr commented Nov 23, 2023

View reviewed changes

alexghr added 2 commits November 23, 2023 14:55

refactor: add loop labels for more elegant breaks

b33ce02

refactor: use of transactions

ca43db3

alexghr marked this pull request as ready for review November 24, 2023 08:28

Merge branch 'master' into alexg/feat/persistent-archiver-store

8fc557e

alexghr marked this pull request as draft November 24, 2023 15:49

alexghr added 2 commits November 27, 2023 10:13

fix: merkle trees store last globalVariablesHash

8653839

fix: store/restore indexed tree's leaves to db

a4235f9

alexghr marked this pull request as ready for review November 27, 2023 10:14

fix: pass global vars hash

20c5acd

PhilWindle approved these changes Nov 27, 2023

View reviewed changes

alexghr and others added 4 commits November 27, 2023 10:35

refactor: inject archiver store into archiver

5e202ef

refactor: archiver gets L1 block number from store

b0955a4

refactor: use private property for l1BlockNumber

600002b

Merge branch 'master' into alexg/feat/persistent-archiver-store

f724acd

alexghr added 10 commits November 27, 2023 16:10

refactor: segregate tx and contract indexed data

8a8a60b

refactor: add debug logging to lmdb archiver store

beb0102

fix: use uint32 for block keys

ad2093a

fix: process L1->L2 messages just once

4010d8a

refactor: store entry keys as buffers in dup table

9b46e88

Merge remote-tracking branch 'origin/master' into alexg/feat/persiste…

04e7e73

…nt-archiver-store

fix: memory l1tol2 message store is idempotent

4a85244

test: verify archiver store is idempotent

aa07751

docs: update archiver sync doc

1cb5e12

fix: slow tree test more lenient

801fb6f

alexghr force-pushed the alexg/feat/persistent-archiver-store branch from 186e413 to 801fb6f Compare November 28, 2023 11:59

PhilWindle merged commit 4735bde into master Nov 28, 2023
3 checks passed

PhilWindle deleted the alexg/feat/persistent-archiver-store branch November 28, 2023 14:14

AztecBot mentioned this pull request Nov 28, 2023

chore(master): Release 0.16.1 #3435

Merged

Maddiaa0 pushed a commit that referenced this pull request Nov 28, 2023

feat: persistent archiver store (#3410)

9b7f47c

This PR adds a new implementation for an archiver store that's backed by LMDB. See #3361 for details. Fix #3361

alexghr mentioned this pull request Dec 1, 2023

fix: 'gte'&'lte' need follow ASCII character set order when load leaves #3424

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: persistent archiver store #3410

feat: persistent archiver store #3410

alexghr commented Nov 23, 2023 •

edited

Loading

socket-security bot commented Nov 23, 2023 •

edited

Loading

alexghr Nov 23, 2023

alexghr Nov 23, 2023

PhilWindle Nov 24, 2023

AztecBot commented Nov 24, 2023 •

edited

Loading

L2 block published to L1

L2 chain processing

Circuits stats

Miscellaneous

alexghr commented Nov 24, 2023 •

edited

Loading

PhilWindle Nov 24, 2023

PhilWindle Nov 24, 2023

feat: persistent archiver store #3410

feat: persistent archiver store #3410

Conversation

alexghr commented Nov 23, 2023 • edited Loading

socket-security bot commented Nov 23, 2023 • edited Loading

alexghr Nov 23, 2023

Choose a reason for hiding this comment

alexghr Nov 23, 2023

Choose a reason for hiding this comment

PhilWindle Nov 24, 2023

Choose a reason for hiding this comment

AztecBot commented Nov 24, 2023 • edited Loading

Benchmark results

L2 block published to L1

L2 chain processing

Circuits stats

Miscellaneous

alexghr commented Nov 24, 2023 • edited Loading

PhilWindle Nov 24, 2023

Choose a reason for hiding this comment

PhilWindle Nov 24, 2023

Choose a reason for hiding this comment

alexghr commented Nov 23, 2023 •

edited

Loading

socket-security bot commented Nov 23, 2023 •

edited

Loading

AztecBot commented Nov 24, 2023 •

edited

Loading

alexghr commented Nov 24, 2023 •

edited

Loading