Upgrade libp2p to 0.52.4 #1631

altonen · 2023-09-19T09:09:17Z

Upgrade libp2p to 0.52.4, including a fix:

Set Kademlia to server mode ([network] fix: always set Kademlia mode to server substrate#14703)

TODO

Fix 3 zombienet tests failing:
- zombienet-substrate-0002-validators-warp-sync
- ~~zombienet-polkadot-functional-0005-parachains-disputes-past-session~~ The test is also flaky in other PRs and is not required for CI to succeed.
- zombienet-polkadot-functional-0009-approval-voting-coalescing
Uncomment and update to the actual libp2p API tests in substrate/client/network/src/protocol/notifications/handler.rs.
When upgrading multihash crate as part of libp2p upgrade to version v0.19.1, uncomment the conversion code at

polkadot-sdk/substrate/client/network/types/src/multihash.rs

Line 159 in 7547c49

// TODO: uncomment this after upgrading `multihash` crate to v0.19.1.
Perform a burn-in.

Co-authored-by: Anton <anton.kalyaev@gmail.com>

Recent optimization in `V1Lazy` caused syncing to stall if if a request was sent to a peer who didn't support the request-response protocol.

Addresses getting removed from Kademlia for completely innocuous reasons such as keep-alive timeouts are causing the node to have a lot of dial failures. The dial failures happen because when `ProtocolController` tries to connect to a node, and when the address is trying to be fetched from from `Discovery`, it returns an empty vector since `Kademlia` no longer holds any addresses for the peer.

altonen · 2023-09-19T09:09:30Z

bot fmt

command-bot · 2023-09-19T09:09:35Z

@altonen https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3740116 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/fmt/fmt.sh". Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 3-85cce569-2ba9-445a-a57d-f7f1e8bde6a5 to cancel this command or bot cancel to cancel all commands in this pull request.

command-bot · 2023-09-19T09:10:16Z

@altonen Command "$PIPELINE_SCRIPTS_DIR/commands/fmt/fmt.sh" has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3740116 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3740116/artifacts/download.

dmitry-markin

Looks good! Could you remind why we had to comment-out the tests in Notifications?

substrate/client/network/src/discovery.rs

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

substrate/client/network/src/behaviour.rs

substrate/client/network/src/discovery.rs

altonen · 2023-09-27T08:29:38Z

@xermicus @athei

This libp2p upgrade updates wat which breaks pallet-contracts tests:

---- wasm::tests::balance stdout ----
thread 'wasm::tests::balance' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: Wast(Error { inner: ErrorInner { text: Some(Text { line: 11, col: 5, snippet: "                (get_local 0)" }), file: None, span: Span { offset: 242 }, kind: Custom("unknown operator or unexpected token") } }) }', substrate/frame/contracts/src/wasm/mod.rs:758:40

This can also be reproduced if cargo update -p wat is run on master. Any tips on what's the best way go about fixing these? I'm not familiar with this code at all.

mxinden

Looking forward for this to land. Ping me if you need any help!

substrate/client/network/src/discovery.rs

substrate/client/network/src/transport.rs

xermicus · 2023-10-12T19:14:19Z

@altonen it appears that the wat crate released a patch version that was actually breaking. In master, wat version 1.0.70 is in the Cargo.lock, which still supports this syntax. Which was changed in version 1.0.72, seemingly only supporting local.get 0

nazar-pc · 2024-06-19T05:44:28Z

This PR is now a security issue due to dalek-cryptography/curve25519-dalek#659

dmitry-markin · 2024-06-19T12:38:49Z

This PR is now a security issue due to dalek-cryptography/curve25519-dalek#659

Thanks for reporting! Bumped curve25519-dalek to the latest patched version (4.1.3).

paritytech-cicd-pr · 2024-06-19T13:08:04Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable 2/3
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6503079

substrate/client/mixnet/src/sync_with_runtime.rs

substrate/client/network/src/behaviour.rs

substrate/client/network/src/service.rs

substrate/client/network/src/discovery.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

lexnv · 2024-06-25T12:05:55Z

substrate/client/network/src/peer_info.rs

+				self.identify.on_swarm_event(FromSwarm::NewExternalAddrCandidate(e));
+
+				// Manually confirm all external address candidates.
+				// TODO: consider adding [AutoNAT protocol](https://docs.rs/libp2p/0.52.3/libp2p/autonat/index.html)


This sounds like a good proposal, do we have an issue for this?

I'm not 100% sure we need it — this would change the polkadot protocol, and at the same time we are able to detect external addresses with the heuristic litep2p uses: if the address is reported by 3 or more peers, it's considered external.

May be we can create an issue for implementing this heuristic with libp2p backend, but I don't think it's a high priority. And we would need to handle Identify protocol events directly instead of using libp2p mechanism of external address detection, if we go this route.

substrate/client/network/src/peer_info.rs

substrate/client/network/src/request_responses.rs

lexnv

LGTM! Nicely done here 🚀

jxs · 2024-06-25T18:02:44Z

great work folks! 🚀 hopefully the upgrade to 0.53 will be easier 😁

…ritytech#4198) This PR introduces custom types / substrate wrappers for `Multiaddr`, `multiaddr::Protocol`, `Multihash`, `ed25519::*` and supplementary types like errors and iterators. This is needed to unblock `libp2p` upgrade PR paritytech#1631 after paritytech#2944 was merged. `libp2p` and `litep2p` currently depend on different versions of `multiaddr` crate, and introduction of this "common ground" types is needed to support independent version upgrades of `multiaddr` and dependent crates in `libp2p` & `litep2p`. While being just convenient to not tie versions of `libp2p` & `litep2p` dependencies together, it's currently not even possible to keep `libp2p` & `litep2p` dependencies updated to the same versions as `multiaddr` in `libp2p` depends on `libp2p-identity` that we can't include as a dependency of `litep2p`, which has it's own `PeerId` type. In the future, to keep things updated on `litep2p` side, we will likely need to fork `multiaddr` and make it use `litep2p` `PeerId` as a payload of `/p2p/...` protocol. With these changes, common code in substrate uses these custom types, and `litep2p` & `libp2p` backends use corresponding libraries types.

Upgrade libp2p to 0.52.4, including a fix: * Set Kademlia to server mode (paritytech/substrate#14703) ### TODO - [x] Fix 3 zombienet tests failing: - [x] `zombienet-substrate-0002-validators-warp-sync` - [ ] ~`zombienet-polkadot-functional-0005-parachains-disputes-past-session`~ The test is also flaky in other PRs and is not required for CI to succeed. - [x] `zombienet-polkadot-functional-0009-approval-voting-coalescing` - [x] Uncomment and update to the actual libp2p API tests in [`substrate/client/network/src/protocol/notifications/handler.rs`](https://github.com/paritytech/polkadot-sdk/blob/7331f1796f1a0b0e9fb0cd7bf441239ad9664595/substrate/client/network/src/protocol/notifications/handler.rs#L1009). - [x] When upgrading `multihash` crate as part of libp2p upgrade to version v0.19.1, uncomment the conversion code at https://github.com/paritytech/polkadot-sdk/blob/7547c4942a887029c11cbcfd5103f6d8315ab95c/substrate/client/network/types/src/multihash.rs#L159 - [x] Perform a burn-in. --------- Co-authored-by: Anton <anton.kalyaev@gmail.com> Co-authored-by: command-bot <> Co-authored-by: Dmitry Markin <dmitry@markin.tech> Co-authored-by: Bastian Köcher <git@kchr.de>

altonen and others added 4 commits September 19, 2023 10:35

Upgrade libp2p to 0.52.3

13f140e

Co-authored-by: Anton <anton.kalyaev@gmail.com>

Temporarily stop using V1Lazy

01aac35

Recent optimization in `V1Lazy` caused syncing to stall if if a request was sent to a peer who didn't support the request-response protocol.

Set Kademlia to server mode by default

9bc5509

altonen added the T0-node This PR/Issue is related to the topic “node”. label Sep 19, 2023

altonen requested a review from a team September 19, 2023 09:09

".git/.scripts/commands/fmt/fmt.sh"

5a78cb6

dmitry-markin approved these changes Sep 19, 2023

View reviewed changes

substrate/client/network/src/discovery.rs Outdated Show resolved Hide resolved

substrate/client/network/src/discovery.rs Outdated Show resolved Hide resolved

substrate/client/network/src/discovery.rs Outdated Show resolved Hide resolved

dmitry-markin requested a review from a team September 19, 2023 10:19

altonen and others added 2 commits September 19, 2023 15:51

Fix warnings

b7d2041

Update substrate/client/network/src/discovery.rs

26a8b9a

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

lexnv reviewed Sep 19, 2023

View reviewed changes

substrate/client/network/src/behaviour.rs Outdated Show resolved Hide resolved

lexnv reviewed Sep 19, 2023

View reviewed changes

substrate/client/network/src/discovery.rs Outdated Show resolved Hide resolved

lexnv mentioned this pull request Sep 21, 2023

Add sudo_network_unstable_watch paritytech/json-rpc-interface-spec#91

Merged

altonen added 3 commits September 22, 2023 08:23

Merge remote-tracking branch 'origin/master' into altonen-upgrade-libp2p

3afa3bc

Apply review comments

7aea01d

Merge remote-tracking branch 'origin/master' into altonen-upgrade-libp2p

d35cb99

mxinden reviewed Sep 27, 2023

View reviewed changes

substrate/client/network/src/discovery.rs Outdated Show resolved Hide resolved

substrate/client/network/src/transport.rs Outdated Show resolved Hide resolved

mxinden mentioned this pull request Oct 10, 2023

feat(identity): make rand optional libp2p/rust-libp2p#4349

Merged

4 tasks

altonen added 2 commits October 16, 2023 10:51

Merge remote-tracking branch 'origin/master' into altonen-upgrade-libp2p

8293e72

Fix stuff

d5fd70b

altonen requested a review from athei as a code owner October 16, 2023 21:18

altonen requested a review from a team October 16, 2023 21:18

altonen requested a review from koute as a code owner October 16, 2023 21:18

Merge remote-tracking branch 'origin/master' into altonen-upgrade-libp2p

d234dc4

dmitry-markin requested a review from alexggh June 18, 2024 11:27

Bump curve25519-dalek 4.1.2 -> 4.1.3

87784a6

Merge remote-tracking branch 'origin/master' into altonen-upgrade-libp2p

09f4b33

bkchr approved these changes Jun 21, 2024

View reviewed changes

dmitry-markin and others added 5 commits June 21, 2024 14:59

Apply suggestions from code review

1f748b6

Co-authored-by: Bastian Köcher <git@kchr.de>

Revert unrelated changes in mixnet

0f447da

Remove multiaddr dependency from sc-mixnet

01d6649

Remove unneded libp2p dependencies in other crates

bd61aff

Merge remote-tracking branch 'origin/master' into altonen-upgrade-libp2p

d54d484

dmitry-markin requested a review from lexnv June 24, 2024 14:34

minor: rustfmt

6e5b4ef

dmitry-markin mentioned this pull request Jun 24, 2024

Get rid of libp2p dependency in sc-authority-discovery #4859

Closed

lexnv reviewed Jun 25, 2024

View reviewed changes

substrate/client/network/src/peer_info.rs Show resolved Hide resolved

lexnv reviewed Jun 25, 2024

View reviewed changes

substrate/client/network/src/request_responses.rs Show resolved Hide resolved

lexnv approved these changes Jun 25, 2024

View reviewed changes

dmitry-markin added this pull request to the merge queue Jun 25, 2024

Merged via the queue into master with commit 414a8fc Jun 25, 2024
164 of 166 checks passed

dmitry-markin deleted the altonen-upgrade-libp2p branch June 25, 2024 14:19

alexggh mentioned this pull request Aug 2, 2024

network/libp2p: worker CPU usage increases from v1.13.0 to v1.15.0 #5221

Open

This was referenced Aug 21, 2024

Update polkadot-sdk from v1.11.0 to stable2407 moondance-labs/tanssi#659

Closed

Update polkadot-sdk from v1.11.0 to stable2407 moonbeam-foundation/moonbeam#2912

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade libp2p to 0.52.4 #1631

Upgrade libp2p to 0.52.4 #1631

altonen commented Sep 19, 2023 •

edited by dmitry-markin

Loading

altonen commented Sep 19, 2023

command-bot bot commented Sep 19, 2023 •

edited

Loading

command-bot bot commented Sep 19, 2023

dmitry-markin left a comment

altonen commented Sep 27, 2023

mxinden left a comment

xermicus commented Oct 12, 2023

nazar-pc commented Jun 19, 2024

dmitry-markin commented Jun 19, 2024

paritytech-cicd-pr commented Jun 19, 2024

lexnv Jun 25, 2024

dmitry-markin Jun 25, 2024

dmitry-markin Jun 25, 2024

lexnv left a comment

jxs commented Jun 25, 2024

Upgrade libp2p to 0.52.4 #1631

Upgrade libp2p to 0.52.4 #1631

Conversation

altonen commented Sep 19, 2023 • edited by dmitry-markin Loading

TODO

altonen commented Sep 19, 2023

command-bot bot commented Sep 19, 2023 • edited Loading

command-bot bot commented Sep 19, 2023

dmitry-markin left a comment

Choose a reason for hiding this comment

altonen commented Sep 27, 2023

mxinden left a comment

Choose a reason for hiding this comment

xermicus commented Oct 12, 2023

nazar-pc commented Jun 19, 2024

dmitry-markin commented Jun 19, 2024

paritytech-cicd-pr commented Jun 19, 2024

lexnv Jun 25, 2024

Choose a reason for hiding this comment

dmitry-markin Jun 25, 2024

Choose a reason for hiding this comment

dmitry-markin Jun 25, 2024

Choose a reason for hiding this comment

lexnv left a comment

Choose a reason for hiding this comment

jxs commented Jun 25, 2024

altonen commented Sep 19, 2023 •

edited by dmitry-markin

Loading

command-bot bot commented Sep 19, 2023 •

edited

Loading