From b7191d91b79614f91e25830a622ffbdcc7ac2e81 Mon Sep 17 00:00:00 2001 From: Trevor Berrange Sanchez Date: Wed, 10 Jul 2024 20:23:47 +0200 Subject: [PATCH] Add section for snapshot generation to readme --- src/accountsdb/readme.md | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/src/accountsdb/readme.md b/src/accountsdb/readme.md index 157b0f284..315283988 100644 --- a/src/accountsdb/readme.md +++ b/src/accountsdb/readme.md @@ -188,7 +188,7 @@ loading from a snapshot begins in `accounts_db.loadFromSnapshot` is a very expensive operation. the steps include: -- reads and load all the account files +- reads and load all the account files based on the snapshot manifest's file map - validates + indexes every account in each file (in parallel) - combines the results across the threads (also in parallel) @@ -233,6 +233,25 @@ after validating accounts-db data, we also validate a few key structs: - `Bank` : contains `bank_fields` which is in the snapshot metadata (not used right now) - `StatusCache / SlotHistory Sysvar` : additional validation performed in `status_cache.validate` +## generating a snapshot + +*note:* at the time of writing, this functionality is in its infancy. + +The core logic for generating a snapshot lives in `accounts_db.db.writeSnapshotTarWithFields`; the principle entrypoint is `AccountsDB.writeSnapshotTar`. +The procedure consists of writing the version file, the status cache (`snapshots/status_cache`) file, the snapshot manifest (`snapshots/{SLOT}/{SLOT}`), +and the account files (`accounts/{SLOT}.{FILE_ID}`). This is all written to a stream in the TAR archive format. + +The snapshot manifest file content is comprised of the bincoded (bincode-encoded) data structure `SnapshotFields`, which is an aggregate of: +* implicit state: data derived from the current state of AccountsDB, like the file map for all the account which exist at that snapshot, or which have + changed relative to a full snapshot in an incremental one +* configuration state: data that is used to communicate details about the snapshot, like the full slot to which an incremental snapshot is relative. + +A snapshot is determined to be full or incremental based on whether the incremental bank field for snapshot persistence is null or not. + +If the specified snapshot manifest indicates we're generating an incremental snapshot, we only write account files present in AccountsDB which +are exactly equal to the latest (or 'largest' as named in the code) rooted slot. For full snapshots, we write all account files present in AccountsDB +which are rooted - as in, less than or equal to the latest rooted slot. + ## read/write benchmarks `BenchArgs` contains all the configuration of a benchmark (comments describe each parameter) - found at the bottom of `db.zig` @@ -257,4 +276,4 @@ swissmapBenchmark(500k accounts) 1 7715875 7715875 0 WRITE: 17.163ms (1.44x faster than std) READ: 50.975ms (0.70x faster than std) swissmapBenchmark(1m accounts) 1 17163500 17163500 0 17163500 -``` \ No newline at end of file +```