core, light: get rid of the dual mutexes, hard to reason with

[karalabe]

Possibly fixes #18421 and #18388.

The block chain and light chain currently have two mutexes that protect its internal state: mu and chainmu. The mu supposed to protect internal state from getting corrupted whilst chaimu from multiple modifications to the chain. However, modifying the internal state is equivalent to modifying the chain.

Most of our code ended up only locking mu (e.g. Rollback, SetHead), which although weren't chain insertions, they could really screw up the inserter. The reason this didn't blow up earlier is because most of these method calls are serialized by the downloader, hiding the ticking time bomb. Apparently a side chain import PR managed to destabilize the event orders and somehow triggered a chain modification without locking mu.

[zsfelfoldi]

LGTM

[holiman]

Haven't reviewed the code yet, but a fast-sync was completed with this PR without problems on mon07. Full sync still in progress on mon06.

[holiman]

Full sync at 4926430, no problems so far

[holiman]

Full sync at 6032090

[holiman]

Full sync at 6556437

[liuhuanhui]

How can I repair it?

[fedecaccia]

Even with the update, I get the same error.

[ninjaahhh]

just curious, will the following scenario cause a dead lock? though as the comment says, it only happens if the database is corrupted or empty

bc.SetHead -> bc.loadLastState -> bc.Reset -> bc.ResetWithGenesisBlock -> bc.SetHead