Rocks db window service #1888

carllin · 2018-11-22T11:53:00Z

Problem

Current implementation of window service doesn't use new db_window module

Summary of Changes

Integrate RocksDb into window service
Fix associated tests

Fixes #

rob-solana · 2018-11-23T18:20:58Z

src/bin/replicator.rs

    let (replicator, leader_info) = Replicator::new(
+        db_ledger,


I think it'd be better if you passed the ledger_path as the argument, did the opening in src/fullnode.rs. This way we can move to sled without changing this code.

I think this comment was meant for bin/fullnode.rs? I made the suggested change there :)

is the change not appropriate here, as well?

The replicator doesn't open a fullnode, but I can move the open logic to src/replicator.rs to parallel the change in src/fullnode.rs

rob-solana · 2018-11-23T18:21:32Z

src/bin/fullnode.rs

 extern crate solana_metrics;

 use clap::{App, Arg};
+use rocksdb::{Options, DB};


I think we want to encapsulate these types so moving to sled is easier

rob-solana · 2018-11-23T18:22:39Z

src/db_window.rs

@@ -273,6 +274,8 @@ pub fn process_blob(
    let is_coding = blob.read().unwrap().is_coding();

    // Check if the blob is in the range of our known leaders. If not, we return.
+    // TODO: Need to update slot in broadcast, otherwise this check will fail with


how about a github issue instead?

rob-solana · 2018-11-23T18:24:23Z

src/bin/fullnode.rs

 use solana::client::mk_client;
 use solana::cluster_info::{Node, FULLNODE_PORT_RANGE};
+use solana::db_ledger::{write_entries_to_ledger, DB_LEDGER_DIRECTORY};


how come everybody has to pass in DB_LEDGER_DIRECTORY? could db_ledger do that for them, instead?

…node, clean up tests

* Refactor and add metrics: - Combine remove_* and update_* functions to reduce locking on cost-tracker and iteration. - Add method to calculate executed transaction cost by directly using actual execution cost and loaded accounts size; - Wireup histogram to report loaded accounts size; - Report time of block limits checking; - Move account counters from ExecuteDetailsTimings to ExecuteAccountsDetails; * Move committed transactions adjustment into its own function

…s#1888)" This reverts commit c3fadac.

…olana-labs#1888) (solana-labs#1900) * Refactor and additional metrics for cost tracking (solana-labs#1888) * Refactor and add metrics: - Combine remove_* and update_* functions to reduce locking on cost-tracker and iteration. - Add method to calculate executed transaction cost by directly using actual execution cost and loaded accounts size; - Wireup histogram to report loaded accounts size; - Report time of block limits checking; - Move account counters from ExecuteDetailsTimings to ExecuteAccountsDetails; * Move committed transactions adjustment into its own function (cherry picked from commit c3fadac) * rename cost_tracker.account_data_size to better describe its purpose is to tracker per-block new account allocation --------- Co-authored-by: Tao Zhu <82401714+tao-stones@users.noreply.github.com> Co-authored-by: Tao Zhu <tao@solana.com>

…port of solana-labs#1888) (solana-labs#1900)" This reverts commit 0aef62e.

…s#1888)" (solana-labs#1936) This reverts commit c3fadac.

* Refactor and add metrics: - Combine remove_* and update_* functions to reduce locking on cost-tracker and iteration. - Add method to calculate executed transaction cost by directly using actual execution cost and loaded accounts size; - Wireup histogram to report loaded accounts size; - Report time of block limits checking; - Move account counters from ExecuteDetailsTimings to ExecuteAccountsDetails; * Move committed transactions adjustment into its own function

* Refactor and additional metrics for cost tracking (solana-labs#1888) * Refactor and add metrics: - Combine remove_* and update_* functions to reduce locking on cost-tracker and iteration. - Add method to calculate executed transaction cost by directly using actual execution cost and loaded accounts size; - Wireup histogram to report loaded accounts size; - Report time of block limits checking; - Move account counters from ExecuteDetailsTimings to ExecuteAccountsDetails; * Move committed transactions adjustment into its own function * remove histogram for loaded accounts size due to performance impact

…port of solana-labs#1888) (solana-labs#1900) (solana-labs#1937) Revert "v2.0: Refactor and additional metrics for cost tracking (backport of solana-labs#1888) (solana-labs#1900)" This reverts commit 0aef62e.

…abs#1975) * Refactor cost tracking (solana-labs#1954) * Refactor and additional metrics for cost tracking (solana-labs#1888) * Refactor and add metrics: - Combine remove_* and update_* functions to reduce locking on cost-tracker and iteration. - Add method to calculate executed transaction cost by directly using actual execution cost and loaded accounts size; - Wireup histogram to report loaded accounts size; - Report time of block limits checking; - Move account counters from ExecuteDetailsTimings to ExecuteAccountsDetails; * Move committed transactions adjustment into its own function * remove histogram for loaded accounts size due to performance impact (cherry picked from commit f8630a3) * rename cost_tracker.account_data_size to better describe its purpose is to tracker per-block new account allocation --------- Co-authored-by: Tao Zhu <82401714+tao-stones@users.noreply.github.com> Co-authored-by: Tao Zhu <tao@solana.com>

carllin requested a review from rob-solana November 22, 2018 11:53

carllin force-pushed the RocksDbWindowService branch from aceeefb to 5840165 Compare November 22, 2018 12:09

carllin closed this Nov 22, 2018

carllin reopened this Nov 22, 2018

carllin closed this Nov 22, 2018

carllin reopened this Nov 23, 2018

carllin force-pushed the RocksDbWindowService branch 9 times, most recently from 38ce50c to cfd82e4 Compare November 23, 2018 08:15

rob-solana reviewed Nov 23, 2018

View reviewed changes

carllin force-pushed the RocksDbWindowService branch 5 times, most recently from 9bfbef6 to 2a1fb73 Compare November 24, 2018 08:05

carllin mentioned this pull request Nov 24, 2018

Rocks db erasure decoding #1900

Merged

carllin force-pushed the RocksDbWindowService branch 3 times, most recently from 78a94fe to 6116063 Compare November 24, 2018 12:45

rob-solana approved these changes Nov 24, 2018

View reviewed changes

carllin force-pushed the RocksDbWindowService branch from 2075726 to c9ece71 Compare November 25, 2018 00:43

carllin added 10 commits November 24, 2018 17:00

Add db_window module for windowing functions from RocksDb

c9d2b9a

Replace window with db_window functions in window_service

73e4e0a

Fix tests

ad2de74

Make note of change in db_window

7402739

Create RocksDb ledger in bin/fullnode

d3aef10

Make db_ledger functions generic

1a1c627

Add db_ledger to bin/replicator

ac90288

Fix clippy errors

00a8b7e

Use enumerate() where apt

e8b80e4

Encapsulate RocksDb destroy, move opening of RocksDb ledger into full…

bf35019

…node, clean up tests

carllin force-pushed the RocksDbWindowService branch 5 times, most recently from 9718256 to e792b9c Compare November 25, 2018 01:59

Move db_ledger open from bin/replicator to src/replicator

0153f12

carllin force-pushed the RocksDbWindowService branch from e792b9c to 0153f12 Compare November 25, 2018 02:49

carllin merged commit 57a384d into solana-labs:master Nov 25, 2018

carllin mentioned this pull request Nov 26, 2018

Re-enable fixed tests #1907

Merged

rob-solana mentioned this pull request Nov 29, 2018

merge window and ledger #1717

Closed

vkomenda pushed a commit to vkomenda/solana that referenced this pull request Aug 29, 2021

Update some api urls (solana-labs#1888)

34af372

tao-stones added a commit to tao-stones/solana that referenced this pull request Jul 1, 2024

Revert "Refactor and additional metrics for cost tracking (solana-lab…

446ed7a

…s#1888)" This reverts commit c3fadac.

tao-stones added a commit to tao-stones/solana that referenced this pull request Jul 1, 2024

Revert "v2.0: Refactor and additional metrics for cost tracking (back…

5695539

…port of solana-labs#1888) (solana-labs#1900)" This reverts commit 0aef62e.

tao-stones added a commit to tao-stones/solana that referenced this pull request Jul 1, 2024

Revert "Refactor and additional metrics for cost tracking (solana-lab…

d5a84da

…s#1888)" (solana-labs#1936) This reverts commit c3fadac.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rocks db window service #1888

Rocks db window service #1888

carllin commented Nov 22, 2018 •

edited

Loading

rob-solana Nov 23, 2018

carllin Nov 24, 2018 •

edited

Loading

rob-solana Nov 24, 2018

carllin Nov 25, 2018 •

edited

Loading

rob-solana Nov 23, 2018

rob-solana Nov 23, 2018

rob-solana Nov 23, 2018

Rocks db window service #1888

Rocks db window service #1888

Conversation

carllin commented Nov 22, 2018 • edited Loading

Problem

Summary of Changes

rob-solana Nov 23, 2018

Choose a reason for hiding this comment

carllin Nov 24, 2018 • edited Loading

Choose a reason for hiding this comment

rob-solana Nov 24, 2018

Choose a reason for hiding this comment

carllin Nov 25, 2018 • edited Loading

Choose a reason for hiding this comment

rob-solana Nov 23, 2018

Choose a reason for hiding this comment

rob-solana Nov 23, 2018

Choose a reason for hiding this comment

rob-solana Nov 23, 2018

Choose a reason for hiding this comment

carllin commented Nov 22, 2018 •

edited

Loading

carllin Nov 24, 2018 •

edited

Loading

carllin Nov 25, 2018 •

edited

Loading