Invalid PoW verification resulted in an Orphan block on Ethereum Mainnet

[AlexSSD7]

Yesterday, we caught a very odd orphan block. This work package was submitted via eth_submitWork RPC method:

Nonce: 0xc32b010352953f0d
Header Hash: 0xb439887ea92c6838171da6a5bf04d52b8a89f2647d2736aff43a448b1b933273

Mix Digest: 0xb41266b200574a89cb30c8b6839064a47777d216a33b9f6689769639ccbe4d60

------

["0xc32b010352953f0d", "0xb439887ea92c6838171da6a5bf04d52b8a89f2647d2736aff43a448b1b933273", "0xb41266b200574a89cb30c8b6839064a47777d216a33b9f6689769639ccbe4d60"]

Besides validating this PoW solution using our software, I manually checked the work by passing it to the github.com/ethereum/ethash library and github.com/ethereum/go-ethereum/consensus/ethash module (with checking the work with both hashimotoLight and hashimotoFull functions). All of them marked the work package above as valid, having the actual difficulty of 4.5 PH. This should have been a block; however, Geth rejected the work package while printing these logs:

INFO [11-29|17:52:22.078] Commit new mining work                   number=11355065 sealhash="b43988…933273" uncles=0 txs=144   gas=12480527 fees=0.4400674188 elapsed=51.294ms
WARN [11-29|17:52:23.883] Invalid proof-of-work submitted          sealhash="b43988…933273" elapsed="102.806µs" err="invalid mix digest"
INFO [11-29|17:52:25.079] Commit new mining work                   number=11355065 sealhash="1e01a0…b0100e" uncles=0 txs=147   gas=12466945 fees=0.4463130213 elapsed=52.240ms
ERROR[11-29|17:52:25.697] Failed to generate mapped ethash dataset epoch=379 err="mkdir /.ethash: permission denied"
INFO [11-29|17:52:42.772] Commit new mining work                   number=11355065 sealhash="e824f9…bf5f95" uncles=0 txs=147   gas=12466945 fees=0.4564300281 elapsed=1.546s
INFO [11-29|17:52:43.157] Imported new chain segment               blocks=1    txs=137   mgas=12.468   elapsed=273.623ms   mgasps=45.567   number=11355065 hash="5045d4…dc4dcb" dirty=1022.67MiB

System information

Geth version: 1.9.23-stable (via docker image ethereum/client-go)
OS & Version: Linux x86_64
Commit hash: 8c2f271

Linked issues

#21928 (Caught on a different host, but that issue may be related to this one)

[karalabe]

There was an ethash DAG bugfix included in 1.9.24 #21793. Based on the code I was convinced that it can only trigger in January, but perhaps I missed something? Could you check if it reproduces with 1.9.24?

[karalabe]

One odd thing in your logs is err="mkdir /.ethash: permission denied". Did you change file system permissions recently? Or perhaps pointed Geth to a new ethash folder that's not writeable?

[AlexSSD7]

Thank you for the reply, @karalabe. That DAG generation bug should affect epochs only higher than 385. The block's epoch was 378, and I don't think that that bug affected the DAG generation at all.

And about the permission denied error: we use a very strict security policy on running containers. Only the datadir volume was writeable by the geth process. Also, I didn't really know that it was using the ~/.ethash directory. Since the user that was running geth had no home directory, it defaulted to /.ethash, the root volume directory.

Also, the bug that caused the wrong PoW was never seen before and after the orphan block. I see that we were very "lucky" to catch that bug.

[rjl493456442]

Could you please check the submitted sealhash is paired with the pow solution(nonce, mixdigest)? nvm.
I will try to reproduce

[rjl493456442]

It's verified that the submission is valid. The difficulty can be recovered is 4567006823027140.

[holiman]

Closing this, seems stale and a one-off event