Node syncing blocks gets stuck

[shunsukew]

Description
Astar/Shiden node gets stuck syncing blocks at particular block numbers when syncing node from scratch (0 block).
This happens at a certain rate (sometimes happens, sometimes not) and stuck block numbers are usually different.

Deleting entire parachain db ({base path}/chains/astar or {base path}/chains/shiden) works as a workaround, but we need to find out why this syncing error happens in the first place.

Steps to Reproduce

1. Start new node and sync from block number 0

Expected vs. Actual Behavior
Expected
Node can sync to the latest block without an issue

Actual
Block syncing gets stuck sometimes with below error.

nov. 07 13:41:44 Ubuntu-2004-focal-64-minimal astar-collator[1087149]: 2022-11-07 13:41:44 [Parachain] Block import error: Backend error: Can't canonicalize missing block number #2689575 when importing 0xd7c9fa7c54265da3d8d6f2adaddc337099e3674b80b3d7a2c77ffbf83fb4fd70 (#2693671)
nov. 07 13:41:44 Ubuntu-2004-focal-64-minimal astar-collator[1087149]: 2022-11-07 13:41:44 [Parachain] 💔  Error importing block 0xd7c9fa7c54265da3d8d6f2adaddc337099e3674b80b3d7a2c77ffbf83fb4fd70: consensus error: Import failed: Backend error: Can't canonicalize missing block number #2689575 when importing 0xd7c9fa7c54265da3d8d6f2adaddc337099e3674b80b3d7a2c77ffbf83fb4fd70 (#2693671)
nov. 07 13:41:44 Ubuntu-2004-focal-64-minimal astar-collator[1087149]: 2022-11-07 13:41:44 [Parachain] 💔  Error importing block 0x59e5d3f43f8900da9337650b6b0781c4471b718e4a55445f90dd4dfc6d25591b: block has an unknown parent

Environment

• Operating system: Ubuntu 20.04.5 LTS
• Project version/tag: v4.29.1

[shunsukew]

paritytech/substrate#12613

[bkchr]

@shunsukew how easy is that to reproduce? Is it always the same block that fails?

[shunsukew]

It is relatively easy, I'd say it's more difficult to sync without getting this so far.
Block is not the same each time.
I can reproduce it and give ssh access to the node to few devs if you'd like.

[bkchr]

Can you please give me the exact cli flags you are using to reproduce this?

[shunsukew]

This is cli frags using right now.

ExecStart=/usr/local/bin/astar-collator \
  --rpc-cors all \
  --name {NODE_NAME} \
  --base-path /var/lib/astar \
  --pruning archive \
  --execution Wasm \
  --trie-cache-size 0 \
  --rpc-external \
  --ws-external \
  --prometheus-external \
  --rpc-methods Safe \
  --rpc-port 9933 \
  --ws-port 9944 \
  --ws-max-connections 102400 \
  --max-runtime-instances 64 \
  --port 30333 \
  --chain astar \
  --telemetry-url 'wss://telemetry.polkadot.io/submit/ 0'

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[bkchr]

paritytech/substrate#12613

BTW, I fixed this issue now. Or better, worked around it.

[shunsukew]

paritytech/substrate#12613

BTW, I fixed this issue now. Or better, worked around it.

Thank you so much!