Skip to content

Commit

Permalink
Add section for snapshot generation to readme
Browse files Browse the repository at this point in the history
  • Loading branch information
InKryption committed Jul 10, 2024
1 parent 97094e6 commit b7191d9
Showing 1 changed file with 21 additions and 2 deletions.
23 changes: 21 additions & 2 deletions src/accountsdb/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ loading from a snapshot begins in `accounts_db.loadFromSnapshot` is a very
expensive operation.

the steps include:
- reads and load all the account files
- reads and load all the account files based on the snapshot manifest's file map
- validates + indexes every account in each file (in parallel)
- combines the results across the threads (also in parallel)

Expand Down Expand Up @@ -233,6 +233,25 @@ after validating accounts-db data, we also validate a few key structs:
- `Bank` : contains `bank_fields` which is in the snapshot metadata (not used right now)
- `StatusCache / SlotHistory Sysvar` : additional validation performed in `status_cache.validate`

## generating a snapshot

*note:* at the time of writing, this functionality is in its infancy.

The core logic for generating a snapshot lives in `accounts_db.db.writeSnapshotTarWithFields`; the principle entrypoint is `AccountsDB.writeSnapshotTar`.
The procedure consists of writing the version file, the status cache (`snapshots/status_cache`) file, the snapshot manifest (`snapshots/{SLOT}/{SLOT}`),
and the account files (`accounts/{SLOT}.{FILE_ID}`). This is all written to a stream in the TAR archive format.

The snapshot manifest file content is comprised of the bincoded (bincode-encoded) data structure `SnapshotFields`, which is an aggregate of:
* implicit state: data derived from the current state of AccountsDB, like the file map for all the account which exist at that snapshot, or which have
changed relative to a full snapshot in an incremental one
* configuration state: data that is used to communicate details about the snapshot, like the full slot to which an incremental snapshot is relative.

A snapshot is determined to be full or incremental based on whether the incremental bank field for snapshot persistence is null or not.

If the specified snapshot manifest indicates we're generating an incremental snapshot, we only write account files present in AccountsDB which
are exactly equal to the latest (or 'largest' as named in the code) rooted slot. For full snapshots, we write all account files present in AccountsDB
which are rooted - as in, less than or equal to the latest rooted slot.

## read/write benchmarks
`BenchArgs` contains all the configuration of a benchmark (comments describe each parameter)
- found at the bottom of `db.zig`
Expand All @@ -257,4 +276,4 @@ swissmapBenchmark(500k accounts) 1 7715875 7715875 0
WRITE: 17.163ms (1.44x faster than std)
READ: 50.975ms (0.70x faster than std)
swissmapBenchmark(1m accounts) 1 17163500 17163500 0 17163500
```
```

0 comments on commit b7191d9

Please sign in to comment.