Adopt heaviest subtree fork choice rule

[carllin]

Problem

Old fork selection mechanism was hard to reason about and could cause liveness problems when
interacting with switching proofs: #10343 (comment)

Summary of Changes

1. Add fork_weight_tracker.rs to keep track of the best subtree
2. Replace old fork choice rule in replay_stage.rs and consensus.rs

Fixes #

[carllin]

@aeyakovenko thinking through the best way to migrate onto this new fork choice rule.

I think the problem with immediate unlocking this feature is if both rules choose different blocks:

                                   A 
                             /           \
             (old best, 50%)  B            C (new best, 50%)

It seems the safest thing to do is a rolling upgrade, which unlocks the new for choice rule when the root is greater than some slot UNLOCK_FORK_CHOICE. I think the switching threshold shouldn't cause problems during the transition because the new fork choice rule will choose a branch that has more stake/enough stake (38%) for a switch.

                 A (UNLOCK_FORK_CHOICE)
              /      \
            B        C (new best, 62%, even if not locked out past block D, eventually voters will)
           /
         D (old best, 38%)

Sucky part is that means I have to reintroduce the old fork choice logic :P

[codecov]

Codecov Report

Merging #10441 into master will increase coverage by 0.1%.
The diff coverage is 96.3%.

@@            Coverage Diff            @@
##           master   #10441     +/-   ##
=========================================
+ Coverage    81.6%    81.7%   +0.1%     
=========================================
  Files         296      299      +3     
  Lines       69320    69794    +474     
=========================================
+ Hits        56608    57078    +470     
- Misses      12712    12716      +4     

[aeyakovenko]

i think i need you to walk me through this :)

[carllin]

@mvines, fyi, one more of our favorite rolling upgrades 😄 . This one should be done before any of the switching proof features are enabled.

[mvines]

sg, just tag me again when this lands so I can manage the rollout as clusters upgrade to 1.3

[mvines]

@carllin - I removed the v1.2 tag, this appears way to risky for v1.2. This can ride the v1.3 train

[aeyakovenko]

@carllin - I removed the v1.2 tag, this appears way to risky for v1.2. This can ride the v1.3 train

we would really like to test this in 1.2, otherwise the whole optimistic conf train gets pushed back a month.

[mvines]

The code churn in here is huge.

[mvines]

The problem is that optimistic conf basically missed the 1.2 train. It's a feature that didn't come in by the branch date, or even a day or two later. Ramming in a ton of new code weeks after the branch date isn't a good way to stabilize a release.

[mvines]

v1.2 label restored. I'm still quite nervous about enabling this in the v1.2 timeframe, but given that this can be landed disabled I'll 🙈 and 🙏 for the best. We can discuss a timeline for actually enabling on v1.2 once testnet upgrades to v1.2 and is running well