Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot bootstrap tracing node on moonbeam network #1410

Closed
the-nando opened this issue Apr 14, 2022 · 3 comments
Closed

Cannot bootstrap tracing node on moonbeam network #1410

the-nando opened this issue Apr 14, 2022 · 3 comments

Comments

@the-nando
Copy link

I'm trying to bootstrap from scratch a tracing node on the Moonbeam network. I have another node on the Moonriver network on the same infrastructure and that's working fine.
Version: moonbeam 0.22.0-9fa830c7e-x86_64-linux-gnu
CLI args:

/moonbeam/moonbeam --base-path=/data --chain=moonbeam --name='moonbeamnode1' \
  --pruning=archive --state-cache-size=1 --db-cache=40000 --ethapi=debug,trace,txpool \
  --wasm-runtime-overrides=/moonbeam/moonbeam-substitutes-tracing --execution=wasm \
  --runtime-cache-size=32 --prometheus-external --prometheus-port 6060 --rpc-external --rpc-cors all \
  --rpc-methods Unsafe --disable-log-color -l sync=debug -l sub-libp2p=debug \
  -- --execution=wasm --pruning=archive --name='moonbeamnode1 (Embedded Relay)'

After some initial syncing it stops processing with these errors:

2022-04-14 06:21:10.947 DEBUG tokio-runtime-worker sync: [🌗] Peer is on different chain (our genesis: 0xfe58…b76d theirs: 0x70f0…5439)
2022-04-14 06:21:10.947 DEBUG tokio-runtime-worker sync: [🌗] 12D3KooWPVuVHpV51R7f4sYQSSNXyYy7DRosTGyT2ewqjdiiaLeo disconnected
2022-04-14 06:21:11.141 DEBUG tokio-runtime-worker sync: [🌗] Peer is on different chain (our genesis: 0xfe58…b76d theirs: 0x401a…474b)
2022-04-14 06:21:11.141 DEBUG tokio-runtime-worker sync: [🌗] 12D3KooWS4xVSuRVrN2miVBjiWi25S1fctLJ6T92m5yyRzwr5rcs disconnected
2022-04-14 06:21:11.179 DEBUG tokio-runtime-worker sync: [🌗] Peer is on different chain (our genesis: 0xfe58…b76d theirs: 0x6f24…8b5f)
2022-04-14 06:21:11.179 DEBUG tokio-runtime-worker sync: [🌗] 12D3KooWGDu4jG4eDi7zbTK9k8dLsEiEhG6UwBM7K9TUTBmNsFgM disconnected
2022-04-14 06:21:11.780  INFO tokio-runtime-worker substrate: [🌗] ⚙️  Syncing  0.0 bps, target=#815902 (24 peers), best: #799484 (0x2610…e6af), finalized #146726 (0x9b3b…55ab), ⬇ 1.6kiB/s ⬆ 1.6kiB/s
2022-04-14 06:21:28.663 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:21:43.046 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:22:28.663 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:22:43.047 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:23:28.663 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:23:43.047 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:24:28.663 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:24:43.047 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:25:28.663 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:25:43.047 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:26:28.664 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:26:43.047 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:27:28.664 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:27:43.047 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:28:28.664 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:28:43.047 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:29:28.664 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:29:43.048 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:30:28.664 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:30:43.048 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:31:28.664 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:31:43.048 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:32:28.664 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:32:43.048 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:33:28.665 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:33:43.048 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:34:28.665 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:34:43.048 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:35:28.665 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:35:43.049 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:36:28.665 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:36:43.049 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:37:28.665 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:37:43.049 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:38:28.665 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:38:43.049 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:39:28.665 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:39:43.049 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:40:28.666 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:40:43.049 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:41:28.666 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:41:40.160  WARN tokio-runtime-worker libp2p_kad::behaviour: [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"5\xcb\xfe$20\t/Q\xb0[\x84p\x1f\xe6D\xd4\xda\x82\xb8\x92\xb2\0\xea\xceG\x9bL\xdec+\x84"), success: [], quorum: 20 })
2022-04-14 06:41:40.160  WARN tokio-runtime-worker libp2p_kad::behaviour: [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"{\x16\xae\xabb\xc5\xa0\xdb\xd7F\xb2\xe7W\x94\xc3\xe7\x19\xf1\xf7*\x9e\x92|\x96CTP\xb3?\xe8kA"), success: [], quorum: 20 })
2022-04-14 06:41:40.160  WARN tokio-runtime-worker libp2p_kad::behaviour: [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"n\xfe\xc2\x07\xd6\x83\xdaz-\xa9\xb9DWj^\x06\x02^\xb1h\x8c\xe9P\xe0\xbb\xc2>\xa308h\xd1"), success: [], quorum: 20 })
2022-04-14 06:41:40.160  WARN tokio-runtime-worker libp2p_kad::behaviour: [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"w\xbch\xcf\xbd\x0e\xa8\xa4Q\xa3\xa9\x02W\xc2\xd4\xc1\x12\x1c\xaf=3\xc2\xe0\x18\x1cq\xf3\xb2}P\xdd\x11"), success: [], quorum: 20 })
2022-04-14 06:41:40.160  WARN tokio-runtime-worker libp2p_kad::behaviour: [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xe5\xe4\x0c\x08\x95\x06Q\xf5\xa6g\xb4\x94\xc3\xfe\xca\x85#dp\x1a|'<%|E\x8c\x0b\xb1!\t\xce"), success: [], quorum: 20 })
2022-04-14 06:41:40.160  WARN tokio-runtime-worker libp2p_kad::behaviour: [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\x19+,\xcb\xe3\x0e\xd7\xe0\x96O\xde}\x83:\xd6\x91H\xb2\x88Se\x93\xd9\x88\x1c\0 \xf7\xda\xf9\xe3G"), success: [], quorum: 20 })
2022-04-14 06:41:40.160  WARN tokio-runtime-worker libp2p_kad::behaviour: [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xb6t\xc2\xf3\xf4\xd0\xce\xa5_'\x98\xc4\xd5\xc1F\xc6\xc9\xe1(f\xb7\xe0\xff\xb6RkB\\$,\xeb\xb5"), success: [], quorum: 20 })
2022-04-14 06:41:40.160  WARN tokio-runtime-worker libp2p_kad::behaviour: [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"!+\xb1\xd6L\xc4<\x81\xe9I\x91\xe3\xa9X\xfd\xde\xea\x9c,\xe8\xc0\x98\xbb)\xf1\xbf \x8b\x01~\x0f\x05"), success: [], quorum: 20 })
2022-04-14 06:41:40.160  WARN tokio-runtime-worker libp2p_kad::behaviour: [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xccz\xca5\x07\x1a\xc3\x0b\ny\xb4\x06i\xc6\xaan\xe4s\x9d\x9e\x14>\xaaS\xcf[T)\xb7\x9e\x8c\xb9"), success: [], quorum: 20 })
2022-04-14 06:41:43.050 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:42:28.666 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:42:43.050 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:43:28.666 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:43:43.050 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:44:28.666 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)
2022-04-14 06:44:43.050 DEBUG tokio-runtime-worker sub-libp2p: [Relaychain] Kademlia paused due to high number of connections (746)
2022-04-14 06:45:28.666 DEBUG tokio-runtime-worker sub-libp2p: [🌗] Kademlia paused due to high number of connections (122)

Any idea how can I go about troubleshooting this issue? I have tried pruning the parachain's Rocksdb but it eventually encountered the same error.
@crystalin
Copy link
Collaborator

I've never encountered this issue before. I'd suggest to fully delete the data folder completely and retry.
Maybe @eskimor knows about this.

@the-nando
Copy link
Author

@crystalin it turned out to be an issue similar to the one reported here. Increasing net.core.somaxconn eventually allowed me to get the node up and running.

@crystalin
Copy link
Collaborator

Great, thank you for the feedback @gab-nando

@Akagi201 Akagi201 mentioned this issue Jun 6, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@crystalin @the-nando