Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get rid of libp2p dependency in sc-authority-discovery #5842

Open
wants to merge 116 commits into
base: master
Choose a base branch
from

Conversation

ndkazu
Copy link
Contributor

@ndkazu ndkazu commented Sep 26, 2024

Issue

#4859

Description

This PR removes libp2p types in authority-discovery, and replace them with network backend agnostic types from sc-network-types.
The sc-network interface is therefore updated accordingly.

@ndkazu
Copy link
Contributor Author

ndkazu commented Oct 3, 2024

@dmitry-markin , @bkchr , @lexnv , I think we're ready for a review.

@ndkazu
Copy link
Contributor Author

ndkazu commented Oct 5, 2024

It seems that I cannot add new reviewers myself: I don't see the gear usuallly present on the right side of Reviewers

substrate/client/network/types/src/rec.rs Outdated Show resolved Hide resolved
substrate/client/network/types/src/rec.rs Outdated Show resolved Hide resolved
substrate/client/network/types/src/rec.rs Outdated Show resolved Hide resolved
substrate/client/network/types/src/rec.rs Outdated Show resolved Hide resolved
substrate/client/network/types/src/rec.rs Outdated Show resolved Hide resolved
@@ -907,7 +907,7 @@ impl<B: BlockT + 'static, H: ExHashT> NetworkBackend<B, H> for Litep2pNetworkBac
);

self.event_streams.send(Event::Dht(
DhtEvent::ValueNotFound(libp2p::kad::RecordKey::new(&key))
DhtEvent::ValueNotFound(sc_network_types::rec::Key::new::<sc_network_types::rec::Key>(&key.to_vec().into()))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are passing the same type as generic argument?

Copy link
Contributor Author

@ndkazu ndkazu Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The compiler complains that the type of sc_network_types::rec::Key::new cannot be infered, and so passing the type as the generic fixed the problem.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is because of the .into() call, as the compiler does not know what to convert the key.to_vec() into. Just keeping it as key.to_vec() should be enough. Also, the KademliaKey that is needed here is under your control, so you can implement From::<libp2p::kad::RecordKey> for it and use key.into() here instead of sc_network_types::rec::Key::new().

@ndkazu ndkazu requested a review from bkchr October 13, 2024 14:46
Copy link
Contributor

@dmitry-markin dmitry-markin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going in the right direction! To completely get rid of libp2p types we must ensure no nested types in sc-authority-discovery are libp2p types. After this is done, no .into() and explicit template arguments should be needed.

@@ -580,7 +580,7 @@ where

debug!(target: LOG_TARGET, "Value for hash '{:?}' found on Dht.", v.record.key);

if let Err(e) = self.handle_dht_value_found_event(v) {
if let Err(e) = self.handle_dht_value_found_event(v.into()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What we want is, actually, to get in authority-discovery types without nested libp2p types. In this case, DhtEvent should be updated to carry substrate-specific types and not wrap libp2p (neither litep2p) types. So, no .into() should be needed in this line.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will also need to update self.network.put_value(key, value) to use network backend agnostic types all the way down to NetworkWorker (aka "libp2p network backend") / Litep2pNetworkBackend, where this type will be converted into a specific network backend type.

Copy link
Contributor Author

@ndkazu ndkazu Nov 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dmitry-markin , sorry! the into() here was not needed. I made it so that substrate/client/network/src/types.rs does not use libp2p at all anymore. DhtEvent should also be free of any use oflibp2p.
This Leaves us with Litep2pNetworkBackend which is still using libp2p. I am planning to start working on substrate/client/network/src/litep2p/mod.rs and all related files to remove libp2p. I just need you to confirm or correct my understanding of the current status before that 😃

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! There should be no need for libp2p kademlia types in substrate/client/network/src/litep2p/mod.rs now when you have kad.rs with network backend agnostic types.

Ideally, we should get rid of all libp2p types in litep2p network backend, but better to do it as a follow-up PR.

substrate/client/authority-discovery/src/worker.rs Outdated Show resolved Hide resolved
substrate/client/network/types/src/rec.rs Outdated Show resolved Hide resolved
@dmitry-markin
Copy link
Contributor

@ndkazu thanks for looking into this issue and sorry for slow reviews on my side.

@ndkazu
Copy link
Contributor Author

ndkazu commented Nov 5, 2024

@dmitry-markin , what about litep2p ? keep it or remove/replace it?
I see the following import:
litep2p::protocol::libp2p::bitswap::{ BitswapEvent, BitswapHandle, BlockPresenceType, Config, ResponseType, WantType, };

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T0-node This PR/Issue is related to the topic “node”.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants