Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Protocol command streams have been shut down #3496

Closed
tugytur opened this issue Feb 27, 2024 · 6 comments · Fixed by #3523
Closed

Protocol command streams have been shut down #3496

tugytur opened this issue Feb 27, 2024 · 6 comments · Fixed by #3523
Assignees
@dmitry-markin
Labels
I2-bug The node fails to follow expected behavior.

Comments

@tugytur
Copy link
Contributor

tugytur commented Feb 27, 2024

I've updated our westmint bootnode to release 1.8.0 and the logs get flooded with the following log

[Parachain] Protocol command streams have been shut down
[Parachain] Protocol command streams have been shut down
[Parachain] Protocol command streams have been shut down
[Parachain] Protocol command streams have been shut down
[Parachain] Protocol command streams have been shut down
[Parachain] Protocol command streams have been shut down
[Parachain] Protocol command streams have been shut down
[Parachain] Protocol command streams have been shut down
[Parachain] Protocol command streams have been shut down
[Parachain] Protocol command streams have been shut down
@bkchr
Copy link
Member

bkchr commented Feb 27, 2024

Cc @paritytech/networking

@altonen
Copy link
Contributor

altonen commented Feb 28, 2024

Are you able to capture 1-2 minutes of logs before this spam starts?

@tugytur
Copy link
Contributor Author

tugytur commented Feb 28, 2024

Are you able to capture 1-2 minutes of logs before this spam starts?

It starts immediately.


17:04:31 [Parachain] 📦 Highest known block at #6447227
17:04:31 [Parachain] 〽️ Prometheus exporter started at 127.0.0.1:9615
17:04:31 [Parachain] Running JSON-RPC server: addr=127.0.0.1:9944, allowed origins=["*"]
17:04:31 [Relaychain] Sending fatal alert BadCertificate
17:04:31 [Relaychain] 🔍 Discovered new external address for our node: /ip4/213.167.226.186/tcp/30334/ws/p2p/12D3KooWSHLUgqsLVQUbPWPvDvhFSv89yPQDEZ7UaSjU19FYFKbq
17:04:32 [Parachain] 🔍 Discovered new external address for our node: /ip4/213.167.226.186/tcp/30339/ws/p2p/12D3KooWNjKeaANaeZxBAPctmx8jugSYzuw4vnSCJmEDPB5mtRd6
17:04:32 [Parachain] Report 12D3KooWBz8YQiSzZAfFh5U8jvTrAd7ga9JnEfNfskdk371dYcBW: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.
17:04:32 [Parachain] Report 12D3KooWSftdah4a5JuCiyi1BcgJVaiTmNHrvJwtGsuViVYejZZR: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.
17:04:32 [Parachain] 💔 The bootnode you want to connect to at `/dns/boot.metaspan.io/tcp/36052/p2p/12D3KooWBCqfNb6Y39DXTr4UBWXyjuS3hcZM1qTbHhDXxF6HkAJJ` provided a different peer ID `12D3KooWMt5d6uXbtuY4jJMMHdPjWUfMuRgg1VVhoiNP9tKj2Rg1` than the one you expect `12D3KooWBCqfNb6Y39DXTr4UBWXyjuS3hcZM1qTbHhDXxF6HkAJJ`.
17:04:33 [Parachain] Report 12D3KooWLDXDMnnKBZb91zfyFJ7HYmq83T47aCXB262MrfcFU6uh: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.
17:04:34 [Parachain] Report 12D3KooWS4YsJyqj2cqb2GYMpsB1Hgn6AGpWsN1DCG9npb2AjXJw: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.
17:04:34 [Parachain] Report 12D3KooWHiBmHajp8vukqLc2GnSyXcNDHkSHv17aCpAabLgKgUm9: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.
17:04:34 [Parachain] 🔍 Discovered new external address for our node: /dns/westmint.bootnode.amforc.com/tcp/30339/p2p/12D3KooWNjKeaANaeZxBAPctmx8jugSYzuw4vnSCJmEDPB5mtRd6
17:04:34 [Parachain] 🔍 Discovered new external address for our node: /dns/westmint.bootnode.amforc.com/tcp/30339/ws/p2p/12D3KooWNjKeaANaeZxBAPctmx8jugSYzuw4vnSCJmEDPB5mtRd6
17:04:35 [Parachain] Report 12D3KooWCHm1Md7yj5T6kLDGy99n3E6ANaGyAriveuFxoXAExcFu: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.
17:04:36 [Relaychain] Received imported block via RPC: #19734610 (0x25f7…b2a6 -> 0x72f6…b0d5)
17:04:36 [Parachain] 🎉 Received target parachain header #6447226 (0x49fe…8be1) from the relay chain.
17:04:36 [Parachain] Cannot set warp sync target block: no warp sync strategy is active.
17:04:36 [Parachain] Failed to set warp sync target block header, terminating `SyncingEngine`.
17:04:36 [Parachain] Protocol command streams have been shut down
17:04:36 [Parachain] Protocol command streams have been shut down
17:04:36 [Parachain] Protocol command streams have been shut down
17:04:36 [Parachain] Protocol command streams have been shut down
17:04:36 [Parachain] Protocol command streams have been shut down
17:04:36 [Parachain] Protocol command streams have been shut down
17:04:36 [Parachain] Protocol command streams have been shut down
17:04:36 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Relaychain] Received finalized block via RPC: #19734607 (0xa206…29f2 -> 0xe0ea…a612)
17:04:37 [Parachain] ✨ Imported #6447227 (0xa037…3879)
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Ran out of free WASM instances
17:04:37 [Parachain] Ran out of free WASM instances
17:04:37 [Parachain] Ran out of free WASM instances
17:04:37 [Parachain] Ran out of free WASM instances
17:04:37 [Parachain] Ran out of free WASM instances
17:04:37 [Parachain] Ran out of free WASM instances
17:04:37 [Parachain] Ran out of free WASM instances
17:04:37 [Parachain] Ran out of free WASM instances
17:04:37 [Parachain] ♻️  Reorg on #6447227,0x37d7…74c9 to #6447228,0xc29a…1afa, common ancestor #6447226,0x49fe…8be1
17:04:37 [Parachain] ✨ Imported #6447228 (0xc29a…1afa)
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] ✨ Imported #6447229 (0x0628…e8f0)
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:37 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:38 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:39 [Parachain] Protocol command streams have been shut down
17:04:40 [Parachain] Protocol command streams have been shut down
17:04:40 [Parachain] Protocol command streams have been shut down
17:04:41 [Parachain] Protocol command streams have been shut down
17:04:41 [Parachain] Protocol command streams have been shut down
17:04:41 [Parachain] Protocol command streams have been shut down
17:04:41 [Relaychain] Received finalized block via RPC: #19734608 (0xe0ea…a612 -> 0xe221…e105)
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Relaychain] Received imported block via RPC: #19734611 (0x72f6…b0d5 -> 0x505e…9490)
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down
17:04:42 [Parachain] Protocol command streams have been shut down

@dmitry-markin
Copy link
Contributor

This is the real problem causing it:

17:04:36 [Parachain] Cannot set warp sync target block: no warp sync strategy is active.
17:04:36 [Parachain] Failed to set warp sync target block header, terminating `SyncingEngine`.

I'll need to look into it, some bug with split syncing strategies likely.

@dmitry-markin
Copy link
Contributor

dmitry-markin commented Feb 28, 2024 •
edited
Loading

@tugytur could you share how do you start the node?

@tugytur
Copy link
Contributor Author

tugytur commented Feb 28, 2024

@tugytur could you share how do you start the node?

ExecStart=/usr/local/bin/polkadot-parachain \
        --name="Amforc Bootnode" \
        --pruning=10 \
        --public-addr=/dns/westmint.bootnode.amforc.com/tcp/30333/wss \
        --listen-addr=/ip4/127.0.0.1/tcp/30339/ws \
        --public-addr=/ip4/213.167.226.186/tcp/30339 \
        --listen-addr=/ip4/10.10.150.61/tcp/30339 \
        --chain=westmint \
        --database=paritydb \
        --sync=warp \
        --no-telemetry \
        --no-mdns \
        --no-hardware-benchmarks \
        --no-private-ip \
        --prometheus-port=9615 \
        --rpc-port=9944 \
        --in-peers=6 \
        --out-peers=3 \
        --in-peers-light=50 \
        --max-parallel-downloads=2 \
        --runtime-cache-size=1 \
        --max-runtime-instances=1 \
        --wasm-execution=compiled \
        --rpc-methods=safe \
        --rpc-cors=all \
        --relay-chain-rpc-urls="ws://10.10.153.11:9944" \
        --relay-chain-rpc-urls="ws://10.10.153.12:9944" \
        --base-path=/var/blockchain \
        -- \
        --in-peers=0 \
        --out-peers=0 \
        --max-parallel-downloads=0 \
        --in-peers-light=0 \
        --no-telemetry \
        --no-mdns \
        --no-private-ip \
        --runtime-cache-size=1 \
        --no-hardware-benchmarks

You're right, without the warp flag it works. Had it there by default in case I needed to quickly resync.

@dmitry-markin dmitry-markin mentioned this issue Feb 29, 2024
Merged
@dmitry-markin dmitry-markin added the I2-bug The node fails to follow expected behavior. label Feb 29, 2024
@dmitry-markin dmitry-markin self-assigned this Feb 29, 2024
@dmitry-markin dmitry-markin mentioned this issue Mar 1, 2024
Closed
github-merge-queue bot pushed a commit that referenced this issue Mar 1, 2024
@dmitry-markin
Fix crash of synced parachain node run with --sync=warp (#3523)
a1b57a8 
Fixes #3496.
@github-project-automation github-project-automation bot moved this to Blocked ⛔️ in Networking Mar 1, 2024
@dmitry-markin dmitry-markin mentioned this issue Mar 4, 2024
Open
skunert pushed a commit to skunert/polkadot-sdk that referenced this issue Mar 4, 2024
@dmitry-markin @skunert
Fix crash of synced parachain node run with --sync=warp (paritytech…
7707097 
…#3523)

Fixes paritytech#3496.
bgallois pushed a commit to duniter/duniter-polkadot-sdk that referenced this issue Mar 25, 2024
@dmitry-markin
Fix crash of synced parachain node run with --sync=warp (paritytech…
91f601f 
…#3523)

Fixes paritytech#3496.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dmitry-markin dmitry-markin

Labels
I2-bug The node fails to follow expected behavior.
Projects
Networking
Status: Blocked ⛔️
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix crash of synced parachain node run with --sync=warp paritytech/polkadot-sdk
4 participants
@bkchr @dmitry-markin @tugytur @altonen