Cannot process block at slot 112849

[prestonvanloon]

[2019-10-29 10:31:20] DEBUG forkchoice: Executing state transition on block slot=112849
panic: could not process block from fork choice service: could not execute state transition: could not process block: could not process block header: parent root 0x6a4f34d5192245e2ea5d30ba19860ed433fbcffd3981b4a6a0ca53fc486e5fd9 does not match the latest block header signing root in state 0xfc1c108b23cf44872e9ae2238fe88b3b823d47396f6b2148192a4699496f3fc9

goroutine 221 [running]:
github.com/prysmaticlabs/prysm/beacon-chain/sync/initial-sync.(*InitialSync).Start(0xc000a54030)
        beacon-chain/sync/initial-sync/service.go:106 +0x97d
created by github.com/prysmaticlabs/prysm/shared.(*ServiceRegistry).StartAll
        shared/service_registry.go:44 +0x23e

[prestonvanloon]

I'm getting it at 112848 now...

[2019-10-29 10:49:30]  INFO forkchoice: Executing state transition on block root=0x6a4f34d5... slot=112848
[2019-10-29 10:49:30]  WARN interop: Writing block to disk at /tmp/beacon_block_112848.ssz
[2019-10-29 10:49:30]  WARN interop: Writing state to disk at /tmp/beacon_state_112848.ssz
panic: could not process block from fork choice service: could not execute state transition: validate state root failed, wanted: 0xdcfb61eb96ff94dae76a1b8d7ba1dd919401875b567650fb9415f9f1548a7c5f, received: 0x4f19ccb1bdb649d1d3d1da4985980ae9ca4474628116c4263968d04c4297b72f

goroutine 230 [running]:
github.com/prysmaticlabs/prysm/beacon-chain/sync/initial-sync.(*InitialSync).Start(0xc0006e7e90)
        beacon-chain/sync/initial-sync/service.go:106 +0x97d
created by github.com/prysmaticlabs/prysm/shared.(*ServiceRegistry).StartAll
        shared/service_registry.go:44 +0x23e

failure_at_block_112848.tar.gz

[prestonvanloon]

zcli agrees this is bad transition.

zcli transition blocks --pre beacon_state_112847.ssz --post /dev/null beacon_block_112848.ssz
failed block transition to block beacon_block_112848.ssz
block has invalid state root

[prestonvanloon]

All of the balances are different.

Prod = A
Local = B

diff.log

[prestonvanloon]

At this stage of investigation, we are trying to understand how this happened.

Current idea is that this is caching related. We have disabled the shuffled index cache behind a feature flag for now (#3893) and are continuing to try to understand how it manifested.

[prestonvanloon]

Next steps are to walk through every step of block processing to identify potential non-deterministic inputs / side effects.

[prestonvanloon]

This should be resolved with feature flags on caches. Will monitor closely in next testnet.