Paseo #1789153: Storage root mismatch

[EclesioMeloJunior]

Describe the bug

Gossamer node running on Paseo failed to execute block #1789153, the problem is a storage root mismatch.

[jimjbrettj]

Updated context on this issue:

This failure is associated with the runtime upgrade that occurs on block 1789152, in which the spec is upgraded to 1002004. This is the PR associated with this upgrade. If you go to the logs listed in the comments from the test run, or by clicking here you will see that a new pallet "ImOnline" in created.

After the creation of this pallet, you can see the log "Removed 0 ImOnline keys 🧹" and after debugging gossamer we found that this was not consistent with our behavior and we were removing a key.

This meant one of two things: we were either removing a key we shouldn't or adding a key we shouldn't. I modified our code to avoid adding this problematic key and it resolved our problem, meaning the issue was this key should not be added.

Though we have a temporary fix, I am doing more investigations into the root cause of this to find a sustainable solution.