[Consensus Observer] Add ObservedOrderedBlock to various stores.

[JoshLind]

Note: this PR is built on: #15869

Description

This PR adds a new ObservedOrderedBlock wrapper to consensus observer, with skeleton code. This supports self-contained ordered blocks, and blocks with windows (for execution pool).

The PR offers the following commits:

1. Add the new ObservedOrderedBlock type, and integrate it into the pending block store.
2. Add ObservedOrderedBlock to the ordered block store.
3. Add a new hash index to the pending block store (including tests). This will be used for window dependency checking and traversal.

In the next PR, I'll work on the window dependency traversal logic, and start working on the new buffering and clearing logic.

Testing Plan

New and existing test infrastructure.

[trunk-io]

⏱️ 1h 45m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
test-target-determinator	36m	🟩 ⬜ 🟩 🟩 🟩 (+3 more)
check-dynamic-deps	35m	🟩 🟩 🟩 🟩 🟩 (+7 more)
rust-cargo-deny	19m	🟩 🟩 🟩 🟩 🟩 (+7 more)
general-lints	5m	🟩 🟩 🟩 🟩 🟩 (+7 more)
semgrep/ci	4m	🟩 🟩 🟩 🟩 🟩 (+7 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+7 more)
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+3 more)
permission-check	38s	🟩 🟩 🟩 🟩 🟩 (+7 more)
permission-check	31s	🟩 🟩 🟩 🟩 🟩 (+7 more)
permission-check	21s	🟩 🟩 🟩 🟩 🟩 (+3 more)
determine-docker-build-metadata	17s	🟩 🟩 🟩 🟩 🟩 (+3 more)

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on 3197ade1029f17ef5f5c6b1398ec3aab13d5cba4

two traffics test: inner traffic : committed: 12531.84 txn/s, submitted: 12532.79 txn/s, expired: 0.95 txn/s, latency: 3159.19 ms, (p50: 2700 ms, p70: 2900, p90: 3300 ms, p99: 9300 ms), latency samples: 4764860
two traffics test : committed: 99.97 txn/s, latency: 1654.84 ms, (p50: 1600 ms, p70: 1700, p90: 2000 ms, p99: 2500 ms), latency samples: 1700
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.804, avg: 1.618", "ConsensusProposalToOrdered: max: 0.331, avg: 0.326", "ConsensusOrderedToCommit: max: 0.584, avg: 0.517", "ConsensusProposalToCommit: max: 0.913, avg: 0.843"]
Max non-epoch-change gap was: 1 rounds at version 5194231 (avg 0.00) [limit 4], 1.85s no progress at version 5194231 (avg 0.22s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.76s no progress at version 2147346 (avg 0.76s) [limit 16].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[hariria]

Lovely

[hariria]

Two questions here:

1. Is this struct meant to emulate PendingBlocks on the ConsensusObservor side? If so might be good to add a doc comment at the top of the struct referencing it

2. Does blocks_without_payloads: BTreeMap<(u64, Round), Arc<PendingBlockWithMetadata>> need both the epoch and round as keys? Why not just round?

[hariria]

Also maybe blocks_without_payloads could be renamed to blocks_without_payloads_by_epoch_round or just blocks_without_payloads_by_round if epoch isn't necessary

[JoshLind]

Is this struct meant to emulate PendingBlocks on the ConsensusObservor side?

Hmm... I'm not sure. There seem to be some similarities, but I'm reluctant to say they represent exactly the same thing. Probably best to keep them separate for now. 🤔

Why not just round?

Consensus observer can buffer blocks across epoch changes (so we require both fields). This helps with performance around epoch transitions.

Also maybe blocks_without_payloads could be renamed to blocks_without_payloads_by_epoch_round or just blocks_without_payloads_by_round if epoch isn't necessary

Hmm... the only reason to add by_hash was to disambiguate it from the primary store above. Might just leave it for now.