Invalid state root upgrading validator to 0x02

[twoeths]

Describe the bug

This happened in gnosis devnet

Tested this but on first consolidation same target and source validator it apparently worked fine. However, the erigon-lodestar validator node that proposed the consolidation request transaction rejected all peers and forked. It is stuck on this slot: [link](http://x.x.x.x:8083/slot/68121)
Lodestar shows these errors:
Feb-17 07:11:09.621[chain]           debug: Block error slot=68121, code=BLOCK_ERROR_INVALID_STATE_ROOT, slot=68121, root=0xb8362da32e9ba60bb4e76aa3dae3cf3722404f97be7d674a1ba2d61140bae2f0, expectedRoot=0xb3e18d8a01cb01ac43a0e2fc4d11f6f9298d74ed51ed8520f0597001e4499516
Error: BLOCK_ERROR_INVALID_STATE_ROOT
    at verifyBlocksStateTransitionOnly (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlocksStateTransitionOnly.ts:71:13)
    at BeaconChain.verifyBlocksInEpoch (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlock.ts:114:7)
    at BeaconChain.processBlocks (file:///usr/app/packages/beacon-node/src/chain/blocks/index.ts:74:7)
    at JobItemQueue.runJob (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:102:22)
Feb-17 07:11:09.622[sync]          verbose: Batch process error id=Finalized, startEpoch=4257, status=Processing, code=BLOCK_ERROR_INVALID_STATE_ROOT, slot=68121, root=0xb8362da32e9ba60bb4e76aa3dae3cf3722404f97be7d674a1ba2d61140bae2f0, expectedRoot=0xb3e18d8a01cb01ac43a0e2fc4d11f6f9298d74ed51ed8520f0597001e4499516
Error: BLOCK_ERROR_INVALID_STATE_ROOT
    at verifyBlocksStateTransitionOnly (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlocksStateTransitionOnly.ts:71:13)
    at BeaconChain.verifyBlocksInEpoch (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlock.ts:114:7)
    at BeaconChain.processBlocks (file:///usr/app/packages/beacon-node/src/chain/blocks/index.ts:74:7)
    at JobItemQueue.runJob (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:102:22)

The rest of the nodes are working the fine, even nethermind-lodestar, and the validator seems to have updated to 0x02 withdrawal credentials type. [link](http://x.x.x.x:8083/validator/2000)

Forkmon [link](http://x.x.x.x:8080/)
Slot: [link](http://x.x.x.x:8083/slot/0x79b6cf01f4fa6f3710b3e26f33bdcd7f20a6e8f26a110accd4fc0122b4e7a9a0)

in the above case, the node was able to publish block but then cannot process its published block and stale since then

Expected behavior

no error

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.26.0

[twoeths]

@nflaig was able to get persisted pre state

BeaconState_0xebd93736783bc8a7d3040984deec988a2affee07e96ceb67fbbde21448b2f9b5.ssz.zip

post state

BeaconState_0x4d3ff75ab28361709aeb54131cabf3ba336247be1ea96f2768270b9e2ed1e053.ssz.zip

this is the pre state downloaded from other node

gno_good_state_12663.ssz.zip

[twoeths]

the only difference from preState of a good node, and persisted preState in the "invalid state root" case is the withdrawalCredentials of validator 0, which was the validator that we upgraded to 0x02

it showed that this is the issue

lodestar/packages/state-transition/src/util/electra.ts

Line 24 in 9a2cf74

newWithdrawalCredentials[0] = COMPOUNDING_WITHDRAWAL_PREFIX;