Apply cleanups to solana-core for unified scheduler #4123

ryoqun · 2024-12-15T07:15:12Z

Problem

There are various small changes to support block production by unified scheduler.

Summary of Changes

Do them at once in a single pr, rather than 6 different prs.

Commits are well logically separated.

No functional change, so no test is added.

extracted from #3946, see the pr for the general overview.

ryoqun · 2024-12-16T07:40:07Z

core/src/banking_stage.rs

@@ -803,11 +807,25 @@ impl BankingStage {
    }
 }

+#[cfg_attr(feature = "dev-context-only-utils", qualifiers(pub))]
+pub(crate) fn update_bank_forks_and_poh_recorder_for_new_tpu_bank(


refer to https://github.com/anza-xyz/agave/pull/3946/files#r1886311350 (or #3946 (comment)) for justification of this change.

ryoqun · 2024-12-16T07:41:22Z

core/src/validator.rs

@@ -1409,6 +1410,7 @@ impl Validator {
        let cluster_slots =
            Arc::new(crate::cluster_slots_service::cluster_slots::ClusterSlots::default());

+        let root_bank_cache = RootBankCache::new(bank_forks.clone());


refer to https://github.com/anza-xyz/agave/pull/3946/files#r1886312626 (or #3946 (comment)) for justification of this movement.

ryoqun · 2024-12-16T07:42:06Z

perf/src/packet.rs

+pub type BankingPacketBatch = Arc<Vec<PacketBatch>>;
+pub type BankingPacketReceiver = Receiver<BankingPacketBatch>;


these types are planned to be used by solana-unified-scheduler-pool crate, so they need to be moved higher in the dep graph from solana-core.

ryoqun · 2024-12-16T07:43:28Z

core/src/banking_stage.rs

+    poh_recorder: &RwLock<PohRecorder>,
+    tpu_bank: Bank,
+    track_transaction_indexes: bool,
+) {


not returning BankWithScheduler is intentional because production code (i.e. relevant ReplayingStage) doesn't need it, while making dev call-sites more verbose a bit.

ryoqun · 2024-12-16T13:11:54Z

perf/Cargo.toml

@@ -13,6 +13,7 @@ edition = { workspace = true }
 ahash = { workspace = true }
 bincode = { workspace = true }
 bv = { workspace = true, features = ["serde"] }
+crossbeam-channel = { workspace = true }


kind of arbitrary extra dep addition, but i think it's okay for better code organization. alternatively, we can define BankingPacketBatch/BankingPacketReceiver at solana-unified-scheduler-pool without a single dep graph chagne, but it felt me that it's off-place.

Not a big fan of putting it here. perf seems isolated from any communication, so doesn't seem correct to me to add this just for a type-alias.

Maybe now is a good time to just set up a more (imo) reasonable structure of crates? (doesn't matter THAT much since both banking and sigverify are in core)
A typical pattern that I've seen for crates is something like this:

/ ├── stage1/ ├── communication/ └── stage2/

The communication crate is separate from both the upstream and downstream consumer(s), and they both just need to follow the interface defined in communication. Think it is a step toward separating sigverify & banking code from core as well.

I pondered on naming for awhile. how about solana-core-types as the name of the communication crate?

I acknowledge the awkwardness of putting these into solana-perf and I understand the appeal of cleaner crate organization you suggested.

that said, I think that crate will be very small and kind of limited, in practice. say, we can't put the sibling type alias BankingPacketSender into that crate, because that would pull some impls into the crate (i.e. TracedSender). so, the crate will contain strictly some common set of type aliases basically.

so, i named the crate as -type with that in mind.

if it's really okay for you, i'll create such a crate in real quick.

These would be specific to agave client, so agave- prefix is more appropriate for naming imo.
If it's just going to be communication between SigVerifyStage (possibly soon to be renamed VerifyStage) and BankingStage (should we rename this to BlockProductionStage?) I wonder if the name core-types is too generic - there's a LOT of other stuff in core

hmm, how about agave-banking-stage-types, then? i thought we won't rename replaying-stage to block-verification-stage, so picked the banking-stage likewise

I think it's still better to have some sort of comm or something in the name.
banking-stage-types, at least to me, implies they are types internally used in banking stage; which isn't really the case here. They are simply the types used for banking stage ingress.

agave-banking-stage-ingress-types too verbose? terrible naming?

Side-note, I have considered to rename banking-stage to block-production if we break it out into its' own crate.

agave-banking-stage-ingress-types too verbose? terrible naming?

thanks for the suggestion. i went with the name: 2d7807f

Side-note, I have considered to rename banking-stage to block-production if we break it out into its' own crate.

call me old-fashioned. but, i'd like to stick to the naming convention of ***Stage.

ryoqun · 2024-12-16T14:39:17Z

@apfitzge thanks for quick review. Other than #4123 (comment), is there other concerns? If other changes are mergeable, I'll back-off the commit and create another pr for it, while landing other changes in this pr, if possible..

apfitzge · 2024-12-16T15:29:41Z

@apfitzge thanks for quick review. Other than #4123 (comment), is there other concerns? If other changes are mergeable, I'll back-off the commit and create another pr for it, while landing other changes in this pr, if possible..

Still looking, have some other stuff need to review this morning, and didn't get a chance to finish the review of this one yet

apfitzge

Added a few comments here and in the linked comments for justification of changes.

ryoqun · 2025-01-06T14:04:23Z

Added a few comments here and in the linked comments for justification of changes.

@apfitzge I've replied to them respectively: (1) #3946 (comment) and (2) #3946 (comment)

Assuming you're generally ack-ing with the new approach in the (2) comment, I also removed now-needless commit (736b2d0) from this pr.

apfitzge · 2025-01-08T16:00:58Z

banking-stage-ingress-types/Cargo.toml

+homepage = { workspace = true }
+license = { workspace = true }
+edition = { workspace = true }
+


These are basically internal types (at least for now); I'm not sure we should publish this crate, wdyt?
i.e. add publish = false here

Maybe its' fine since it would make it easier for someone to write their own banking stage replacement (good)

apfitzge

lgtm - let's get the crate name reserved then we can merge

ryoqun · 2025-01-09T13:23:58Z

lgtm - let's get the crate name reserved then we can merge

thanks for the review! now, the crate is reserved: https://crates.io/crates/agave-banking-stage-ingress-types

ryoqun force-pushed the unified-scheduler-core-cleanups branch from f6fc326 to f7c63e0 Compare December 15, 2024 07:22

ryoqun changed the title ~~Unified scheduler core cleanups~~ Apply various cleanups to solana-core for unified scheduler Dec 16, 2024

ryoqun changed the title ~~Apply various cleanups to solana-core for unified scheduler~~ Apply cleanups to solana-core for unified scheduler Dec 16, 2024

ryoqun commented Dec 16, 2024

View reviewed changes

ryoqun requested a review from apfitzge December 16, 2024 07:43

ryoqun mentioned this pull request Dec 16, 2024

Wire unified scheduler into banking experimentally #3946

Draft

7 tasks

ryoqun force-pushed the unified-scheduler-core-cleanups branch from f7c63e0 to aabd2ac Compare December 16, 2024 13:08

ryoqun commented Dec 16, 2024

View reviewed changes

apfitzge reviewed Dec 16, 2024

View reviewed changes

ryoqun requested a review from apfitzge December 18, 2024 16:15

ryoqun added 5 commits January 6, 2025 21:56

Setup PohRecorder earlier for unified scheduler

05a970f

Use BankWithScheduler in banking-bench for US

91b2377

Move BankingPacket{Batch,Receiver} to new crate

2d7807f

Extract duplicate tpu bank handling code for US

a1bc4b9

Call wait_for_completed_scheduler in banking simulator

5b1b19b

ryoqun force-pushed the unified-scheduler-core-cleanups branch from aabd2ac to 5b1b19b Compare January 6, 2025 13:39

apfitzge reviewed Jan 8, 2025

View reviewed changes

ryoqun requested a review from apfitzge January 9, 2025 13:22

apfitzge approved these changes Jan 9, 2025

View reviewed changes

ryoqun merged commit 6b6a03b into anza-xyz:master Jan 9, 2025
51 checks passed

apfitzge mentioned this pull request Jan 9, 2025

TransactionView: ReceiveAndBuffer #3820

Merged

ryoqun mentioned this pull request Jan 14, 2025

Apply more unified scheduler related core cleanups #4453

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Apply cleanups to solana-core for unified scheduler #4123

Apply cleanups to solana-core for unified scheduler #4123

ryoqun commented Dec 15, 2024 •

edited

Loading

ryoqun Dec 16, 2024 •

edited

Loading

ryoqun Dec 16, 2024 •

edited

Loading

ryoqun Dec 16, 2024 •

edited

Loading

ryoqun Dec 16, 2024

ryoqun Dec 16, 2024

apfitzge Dec 16, 2024

ryoqun Dec 17, 2024

apfitzge Dec 17, 2024

ryoqun Dec 18, 2024

apfitzge Dec 31, 2024

ryoqun Jan 6, 2025

ryoqun commented Dec 16, 2024

apfitzge commented Dec 16, 2024

apfitzge left a comment

ryoqun commented Jan 6, 2025 •

edited

Loading

apfitzge Jan 8, 2025

apfitzge Jan 8, 2025

apfitzge left a comment

ryoqun commented Jan 9, 2025

		pub type BankingPacketBatch = Arc<Vec<PacketBatch>>;
		pub type BankingPacketReceiver = Receiver<BankingPacketBatch>;

Apply cleanups to solana-core for unified scheduler #4123

Apply cleanups to solana-core for unified scheduler #4123

Conversation

ryoqun commented Dec 15, 2024 • edited Loading

Problem

Summary of Changes

ryoqun Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

ryoqun Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

ryoqun Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryoqun commented Dec 16, 2024

apfitzge commented Dec 16, 2024

apfitzge left a comment

Choose a reason for hiding this comment

ryoqun commented Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apfitzge left a comment

Choose a reason for hiding this comment

ryoqun commented Jan 9, 2025

ryoqun commented Dec 15, 2024 •

edited

Loading

ryoqun Dec 16, 2024 •

edited

Loading

ryoqun Dec 16, 2024 •

edited

Loading

ryoqun Dec 16, 2024 •

edited

Loading

ryoqun commented Jan 6, 2025 •

edited

Loading