Prater invalid state root at slot 1225921

[dapplion]

Describe the bug

Our Prater node running with 920 validators on a Contabo M machine got

2021-09-09T20:24:57.724115394Z Sep-09 20:24:57.723 [CHAIN]           error: Block error slot=1225921 code=BLOCK_ERROR_BEACON_CHAIN_ERROR, error={"message":"Invalid state root at slot 1225921, expected=0xffecfdad1d98909720ea8ee156e5b1c6300e3cf8c70c2b5da6b91e63604db81f, actual=0x3da7d33d7ed958f2b7cc8e286ba7146e558fd058ef73e69137e2ca4123371a8e"}
2021-09-09T20:24:57.724160728Z Error: Invalid state root at slot 1225921, expected=0xffecfdad1d98909720ea8ee156e5b1c6300e3cf8c70c2b5da6b91e63604db81f, actual=0x3da7d33d7ed958f2b7cc8e286ba7146e558fd058ef73e69137e2ca4123371a8e
2021-09-09T20:24:57.724165216Z     at Object.stateTransition (/usr/app/node_modules/@chainsafe/lodestar-beacon-state-transition/src/allForks/stateTransition.ts:67:13)
2021-09-09T20:24:57.724168863Z     at Object.runStateTransition (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/stateTransition.ts:83:30)
2021-09-09T20:24:57.724172390Z     at processBlocksInEpoch (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/process.ts:108:24)
2021-09-09T20:24:57.724176167Z     at runMicrotasks (<anonymous>)
2021-09-09T20:24:57.724179774Z     at runNextTicks (internal/process/task_queues.js:60:5)
2021-09-09T20:24:57.724183130Z     at processTimers (internal/timers.js:497:9)
2021-09-09T20:24:57.724186737Z     at Object.processBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/process.ts:42:3)
2021-09-09T20:24:57.724190173Z     at processBlockJob (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/processor.ts:66:5)
2021-09-09T20:24:57.724193600Z     at Timeout.JobItemQueue.runJob [as _onTimeout] (/usr/app/node_modules/@chainsafe/lodestar/src/util/queue/itemQueue.ts:85:22)
2021-09-09T20:24:57.724197166Z Error: BLOCK_ERROR_BEACON_CHAIN_ERROR
2021-09-09T20:24:57.724200391Z     at processBlocksInEpoch (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/process.ts:149:11)
2021-09-09T20:24:57.724253821Z     at runMicrotasks (<anonymous>)
2021-09-09T20:24:57.724257558Z     at runNextTicks (internal/process/task_queues.js:60:5)
2021-09-09T20:24:57.724260904Z     at processTimers (internal/timers.js:497:9)
2021-09-09T20:24:57.724264140Z     at Object.processBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/process.ts:42:3)
2021-09-09T20:24:57.724267536Z     at processBlockJob (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/processor.ts:66:5)
2021-09-09T20:24:57.724310736Z     at Timeout.JobItemQueue.runJob [as _onTimeout] (/usr/app/node_modules/@chainsafe/lodestar/src/util/queue/itemQueue.ts:85:22)

Keep in mind that the time in the metrics screenshots are in UTC+2, and the logs are in UTC. So the invalid state root issue stopped sync progress and the "stuck" situation caused an OOM eventually

[twoeths]

it looks like this is the old version

root@vmi623737:~/lodestar# docker images | grep lodestar
chainsafe/lodestar      next        33b8afef5d91   7 days ago    245MB

while this PR was merged 5 days ago #3083

we need to pull the latest docker image and try again to confirm

[twoeths]

closing the issue as it works fine in contabo-7 after pulling the latest docker image