Add wen_restart module #33344

wen-coding · 2023-09-21T07:05:08Z

Problem

Add wen_restart module

Summary of Changes

Implement reading LastVotedForkSlots from blockstore.
Add proto file to record the intermediate results.
Also link wen_restart into validator.
Move recreation of tower outside replay_stage so we can get last_vote.

- Implement reading LastVotedForkSlots from blockstore. - Add proto file to record the intermediate results. - Also link wen_restart into validator. - Move recreation of tower outside replay_stage so we can get last_vote.

codecov · 2023-09-21T08:44:18Z

Codecov Report

Merging #33344 (2a61944) into master (64b3613) will increase coverage by 0.0%.
The diff coverage is 78.6%.

@@           Coverage Diff           @@
##           master   #33344   +/-   ##
=======================================
  Coverage    81.7%    81.7%           
=======================================
  Files         805      807    +2     
  Lines      218162   218256   +94     
=======================================
+ Hits       178411   178493   +82     
- Misses      39751    39763   +12

wen-restart/src/wen_restart.rs

core/src/replay_stage.rs

validator/src/cli.rs

AshwinSekar · 2023-09-21T18:01:18Z

wen-restart/src/wen_restart.rs

+    let last_vote_slot = last_vote.last_voted_slot().unwrap();
+    let mut last_vote_fork = vec![last_vote_slot];
+    let mut slot = last_vote_slot;
+    for _ in 0..MAX_SLOTS_ON_VOTED_FORKS {


do we need to go this deep, can we stop at block_commitment_cache.highest_supermajority_root() ?

if i'm behind the smr my last voted fork info doesn't contribute anything to wen restart anyway right?

That is also an option, I think the current logic is trying to be more straightforward so more fault tolerant. What if there is a bug in code so highest_supermajority_root is wrong?

So when we started the initial design we tried to see what the minimum required info is, we had to send last_vote, and need to send the slots on last voted fork so validators with different progress can sync up again. Other than that, we know an outage is probably less than 9 hours before we intervene, and we avoid relying on all other information. Because software can have bugs, you can't predict what is going wrong in an outage.

Does that make sense?

do you know what the latest oom time is, can we actually survive 9 hours without making a root?

either way it seems alright to play it safe and send all these slots.
was selfishly hoping we could reduce the size so that we could include hashes for the inevitable duplicate block restart 😛

I think in past outages validators lasted maybe 3 to 4 hours without making root, we decided to restart before 9 hours.

Yeah whether we can include hash had been debated, and the first version it looks like hash isn't going to make it. We could always add another set of consensus on bankhash in the future though.

wen-restart/src/wen_restart.rs

t-nelson · 2023-09-21T20:18:16Z

@CriesofCarrots mind giving the protobufs stuff a once over for consistency and build reliability?

CriesofCarrots

Just some stylistic suggestions and nits from me. The proto build script itself looks good.

validator/src/cli.rs

wen-restart/protos/wen_restart.proto

CriesofCarrots · 2023-09-21T23:13:08Z

wen-restart/protos/wen_restart.proto

+        DONE = 6;
+    }
+
+    message InitRecord {


It's not totally clear to me what an InitRecord is. Might be more clear if you spell out Init? I'm not sure if Init is a noun or an adjective (or a verb, I suppose).

Renamed to MyLastVotedForkSlots.

wen-restart/src/wen_restart.rs

wen-restart/build.rs

Co-authored-by: Tyera <teulberg@gmail.com>

CriesofCarrots

r+ final nits. (I only reviewed the proto build, usage, and related)

wen-restart/Cargo.toml

wen-restart/src/wen_restart.rs

Co-authored-by: Tyera <teulberg@gmail.com>

core/src/validator.rs

already larger than MAX_SLOTS_ON_VOTED_FORKS.

validator/src/cli.rs

wen-restart/src/wen_restart.rs

core/src/validator.rs

* Add wen_restart module: - Implement reading LastVotedForkSlots from blockstore. - Add proto file to record the intermediate results. - Also link wen_restart into validator. - Move recreation of tower outside replay_stage so we can get last_vote. * Update lock file. * Fix linter errors. * Fix depencies order. * Update wen_restart explanation and small fixes. * Generate tower outside tvu. * Update validator/src/cli.rs Co-authored-by: Tyera <teulberg@gmail.com> * Update wen-restart/protos/wen_restart.proto Co-authored-by: Tyera <teulberg@gmail.com> * Update wen-restart/build.rs Co-authored-by: Tyera <teulberg@gmail.com> * Update wen-restart/src/wen_restart.rs Co-authored-by: Tyera <teulberg@gmail.com> * Rename proto directory. * Rename InitRecord to MyLastVotedForkSlots, add imports. * Update wen-restart/Cargo.toml Co-authored-by: Tyera <teulberg@gmail.com> * Update wen-restart/src/wen_restart.rs Co-authored-by: Tyera <teulberg@gmail.com> * Move prost-build dependency to project toml. * No need to continue if the distance between slot and last_vote is already larger than MAX_SLOTS_ON_VOTED_FORKS. * Use 16k slots instead of 81k slots, a few more wording changes. * Use AncestorIterator which does the same thing. * Update Cargo.lock * Update Cargo.lock --------- Co-authored-by: Tyera <teulberg@gmail.com>

brooksprumo · 2023-10-09T12:34:11Z

wen-restart/build.rs

+    std::io::Result,
+};
+
+fn main() -> Result<()> {


I believe within this main function you'll want to add

println!("cargo:rerun-if-changed=proto/wen_restart.proto");

This causes the generated files to get regenerated if the protos change.

More info:

https://docs.rs/cargo-emit/latest/cargo_emit/macro.rerun_if_changed.html

https://doc.rust-lang.org/cargo/reference/build-scripts.html#build-scripts

https://rust-lang.github.io/rust-bindgen/tutorial-3.html

Thanks for checking, sending another PR to fix it.

brooksprumo · 2023-10-09T12:36:48Z

wen-restart/build.rs

+    // Copied and adapted from
+    // https://github.com/Kimundi/rustc-version-rs/blob/1d692a965f4e48a8cb72e82cda953107c0d22f47/README.md#example
+    // Licensed under Apache-2.0 + MIT
+    match version_meta().unwrap().channel {
+        Channel::Stable => {
+            println!("cargo:rustc-cfg=RUSTC_WITHOUT_SPECIALIZATION");
+        }
+        Channel::Beta => {
+            println!("cargo:rustc-cfg=RUSTC_WITHOUT_SPECIALIZATION");
+        }
+        Channel::Nightly => {
+            println!("cargo:rustc-cfg=RUSTC_WITH_SPECIALIZATION");
+        }
+        Channel::Dev => {
+            println!("cargo:rustc-cfg=RUSTC_WITH_SPECIALIZATION");
+            // See https://github.com/solana-labs/solana/issues/11055
+            // We may be running the custom `rust-bpf-builder` toolchain,
+            // which currently needs `#![feature(proc_macro_hygiene)]` to
+            // be applied.
+            println!("cargo:rustc-cfg=RUSTC_NEEDS_PROC_MACRO_HYGIENE");
+        }
+    }


Looking at the wen-restart module, I'm not seeing anything that requires specialization. Is this block needed? (Usually I see this code to enable the AbiExample testing macros.)

Probably not, I'll send you another PR to remove it.

steviez · 2023-10-10T19:54:34Z

validator/src/cli.rs

@@ -1381,6 +1381,35 @@ pub fn app<'a>(version: &'a str, default_args: &'a DefaultArgs) -> App<'a, 'a> {
                .possible_values(BlockProductionMethod::cli_names())
                .help(BlockProductionMethod::cli_message())
        )
+        .arg(
+            Arg::with_name("wen_restart")


Took a quick look at this because of #33633. I believe (but could be mistaken) that this feature is not yet production ready. If that is correct, i think we should add this

.hidden(hidden_unless_forced())

with the idea being to keep the in-development features hidden from solana-validator --help until they're ready

Add wen_restart module:

228cf2b

- Implement reading LastVotedForkSlots from blockstore. - Add proto file to record the intermediate results. - Also link wen_restart into validator. - Move recreation of tower outside replay_stage so we can get last_vote.

wen-coding requested review from t-nelson, carllin and AshwinSekar September 21, 2023 07:05

wen-coding and others added 4 commits September 21, 2023 00:12

Update lock file.

d575ff8

Merge branch 'master' into wen_restart_add_module

a1fba4e

Fix linter errors.

652f724

Fix depencies order.

d9e2f93

AshwinSekar reviewed Sep 21, 2023

View reviewed changes

wen-restart/src/wen_restart.rs Outdated Show resolved Hide resolved

core/src/replay_stage.rs Outdated Show resolved Hide resolved

validator/src/cli.rs Show resolved Hide resolved

AshwinSekar reviewed Sep 21, 2023

View reviewed changes

wen-restart/src/wen_restart.rs Outdated Show resolved Hide resolved

wen-coding and others added 2 commits September 21, 2023 12:33

Update wen_restart explanation and small fixes.

baebed8

Merge branch 'master' into wen_restart_add_module

9c0c78c

t-nelson requested a review from CriesofCarrots September 21, 2023 20:17

wen-coding removed the request for review from CriesofCarrots September 21, 2023 20:18

wen-coding requested a review from CriesofCarrots September 21, 2023 20:19

Generate tower outside tvu.

57c9228

CriesofCarrots reviewed Sep 21, 2023

View reviewed changes

wen-coding and others added 6 commits September 21, 2023 16:31

Update validator/src/cli.rs

3ac0a02

Co-authored-by: Tyera <teulberg@gmail.com>

Update wen-restart/protos/wen_restart.proto

fd7e157

Co-authored-by: Tyera <teulberg@gmail.com>

Update wen-restart/build.rs

84ee985

Co-authored-by: Tyera <teulberg@gmail.com>

Update wen-restart/src/wen_restart.rs

690cef7

Co-authored-by: Tyera <teulberg@gmail.com>

Rename proto directory.

8c0c04a

Rename InitRecord to MyLastVotedForkSlots, add imports.

bcf9942

CriesofCarrots previously approved these changes Sep 22, 2023

View reviewed changes

wen-restart/Cargo.toml Outdated Show resolved Hide resolved

wen-restart/Cargo.toml Outdated Show resolved Hide resolved

wen-restart/src/wen_restart.rs Outdated Show resolved Hide resolved

wen-coding dismissed CriesofCarrots’s stale review via d5375bd September 22, 2023 03:24

wen-coding and others added 2 commits September 21, 2023 20:24

Update wen-restart/Cargo.toml

d5375bd

Co-authored-by: Tyera <teulberg@gmail.com>

Update wen-restart/src/wen_restart.rs

ab88c0a

Co-authored-by: Tyera <teulberg@gmail.com>

wen-coding and others added 2 commits September 21, 2023 20:30

Move prost-build dependency to project toml.

6c5c8ec

Merge branch 'master' into wen_restart_add_module

fbb920e

wen-coding self-assigned this Sep 22, 2023

AshwinSekar reviewed Sep 26, 2023

View reviewed changes

core/src/validator.rs Show resolved Hide resolved

wen-coding and others added 2 commits October 1, 2023 22:30

No need to continue if the distance between slot and last_vote is

72ada79

already larger than MAX_SLOTS_ON_VOTED_FORKS.

Merge branch 'master' into wen_restart_add_module

d9c76f2

carllin reviewed Oct 6, 2023

View reviewed changes

wen-coding and others added 5 commits October 5, 2023 22:14

Use 16k slots instead of 81k slots, a few more wording changes.

caea7d7

Use AncestorIterator which does the same thing.

e1fb692

Merge branch 'master' into wen_restart_add_module

e3ea405

Update Cargo.lock

433a18c

Update Cargo.lock

2a61944

carllin approved these changes Oct 6, 2023

View reviewed changes

wen-coding merged commit 630feed into solana-labs:master Oct 6, 2023
27 checks passed

wen-coding deleted the wen_restart_add_module branch October 6, 2023 22:13

brooksprumo reviewed Oct 9, 2023

View reviewed changes

wen-coding restored the wen_restart_add_module branch October 9, 2023 14:23

steviez reviewed Oct 10, 2023

View reviewed changes

willhickey mentioned this pull request Mar 28, 2024

v1.18 commits - please ignore anza-xyz/agave#475

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add wen_restart module #33344

Add wen_restart module #33344

wen-coding commented Sep 21, 2023

codecov bot commented Sep 21, 2023 •

edited

Loading

AshwinSekar Sep 21, 2023

wen-coding Sep 21, 2023

AshwinSekar Sep 21, 2023

wen-coding Sep 21, 2023

t-nelson commented Sep 21, 2023

CriesofCarrots left a comment

CriesofCarrots Sep 21, 2023

wen-coding Sep 21, 2023

CriesofCarrots left a comment

brooksprumo Oct 9, 2023

wen-coding Oct 9, 2023

brooksprumo Oct 9, 2023

wen-coding Oct 9, 2023

steviez Oct 10, 2023

Add wen_restart module #33344

Add wen_restart module #33344

Conversation

wen-coding commented Sep 21, 2023

Problem

Summary of Changes

codecov bot commented Sep 21, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t-nelson commented Sep 21, 2023

CriesofCarrots left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CriesofCarrots left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 21, 2023 •

edited

Loading