Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(benchmarks): implement benchmarking for snapshot validation #289

Closed
wants to merge 22 commits into from

Conversation

Rexicon226
Copy link
Contributor

@Rexicon226 Rexicon226 commented Sep 30, 2024

closes #283

i'd like a second opinion on 44d9939. i've found it pretty common that i want to build a specific part of sig and have it output as a binary.
Now it looks like this:

zig build // builds everything and installs everything
zig build benchmark // builds benchmark and runs it
zig build benchmark -Dno-run // builds benchmark and installs it

x19's notes:

example benchmark results (multiple values)

benchmark, read_time_min, read_time_max, read_time_mean, read_time_variance, benchmark, write_time_min, write_time_max, write_time_mean, write_time_variance, 
readWriteAccounts(100k accounts (1_slot - ram index - ram accounts)), 172156041, 158767959, 162868245, 15183799545214, 303852750, 286908417, 292925858, 39820330697776, 
readWriteAccounts(100k accounts (1_slot - disk index - ram accounts)), 165480250, 156170500, 160821658, 7611019088428, 319935833, 286708833, 304248199, 113169780175088,

NOTE: for multiple value outputs its not human readable at all - but i think thats ok -- most of our confirmations of speed ups/slow downs shouldnt be human checked, it should be computed and the increase/decrease should be human checked (which this PR allows for you to do easily)

@Rexicon226 Rexicon226 force-pushed the Rexicon226/snapshot-bench branch from 17db117 to f74f43e Compare September 30, 2024 19:15
@Rexicon226 Rexicon226 changed the title implement benchmarking for snapshot validation feat(benchmarks): implement benchmarking for snapshot validation Sep 30, 2024
@Rexicon226 Rexicon226 marked this pull request as draft October 1, 2024 00:29
@Rexicon226 Rexicon226 marked this pull request as ready for review October 1, 2024 00:31
@0xNineteen 0xNineteen self-requested a review October 1, 2024 20:33
build.zig Show resolved Hide resolved
src/accountsdb/db.zig Outdated Show resolved Hide resolved
src/accountsdb/accounts_file.zig Show resolved Hide resolved
src/utils/fmt.zig Show resolved Hide resolved
src/utils/directory.zig Show resolved Hide resolved
src/utils/directory.zig Outdated Show resolved Hide resolved
src/utils/directory.zig Show resolved Hide resolved
src/accountsdb/db.zig Outdated Show resolved Hide resolved
src/accountsdb/db.zig Show resolved Hide resolved
src/accountsdb/download.zig Show resolved Hide resolved
src/accountsdb/download.zig Outdated Show resolved Hide resolved
src/accountsdb/download.zig Show resolved Hide resolved
@Rexicon226 Rexicon226 force-pushed the Rexicon226/snapshot-bench branch from e46b192 to 696c4fd Compare October 8, 2024 14:07
@0xNineteen
Copy link
Contributor

just to have it written down:

we chatted offline and decided to :

  • modify the benchmarks to support multiple return values
  • add an option to output the results in a more csv-friendly format to make comparisons across different runs easier (can use google sheets to copy and past the output with a new column = columnA - columnB to get the change in performance easier (compared to manually doing it which is error prone and not a good use of time), eg:
    // pub const BenchResult = struct {
    //     BenchmarkAccountsDB.read_time: usize,
    //     BenchmarkAccountsDB.write_time: usize,
    // };

    // '--csv' flag:
    // benchmark name, write_time, read_time
    // readWriteAccounts(100k accounts (1_slot - ram index - disk accounts)), 35.868, 32.22,

@0xNineteen 0xNineteen self-assigned this Oct 15, 2024
@0xNineteen 0xNineteen force-pushed the Rexicon226/snapshot-bench branch 2 times, most recently from c02f47d to 2fa73ed Compare October 15, 2024 18:44
@0xNineteen 0xNineteen requested a review from dnut October 15, 2024 18:54
@0xNineteen 0xNineteen force-pushed the Rexicon226/snapshot-bench branch from 5088c80 to 8abb69f Compare October 16, 2024 16:06
@0xNineteen
Copy link
Contributor

closing to move progress to new branch

@0xNineteen 0xNineteen closed this Oct 17, 2024
@0xNineteen 0xNineteen deleted the Rexicon226/snapshot-bench branch October 17, 2024 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

test(accountsdb): benchmark snapshot loading and validation
3 participants