Proposal: turn the simulation invariants into a validator-assertable option (or maybe even mandatory)

[cwgoes]

In the spirit of pragmatic solutions to provide higher assurance of correctness relatively quickly, I suggest we consider adding a --assert-invariants mode to gaiad (perhaps even enabled by default). When run in this mode, gaiad asserts all the specified invariants at the end of each block and panics if any fail.

This is easy to implement since we can just reuse the existing invariants from the simulation (and any future invariants which might be added to the simulation) and adds a reasonable degree of assurance from certain classes of bugs - if at least 1/3 of stake runs with this invariant assertion mode enabled, and an invariant breaks, the chain will safely halt and we can figure out what's going on & fix it.

This is not a long-term solution as it isn't necessarily incentive-compatible (extra compute by validators which they aren't getting paid for) - but in the short term I think enough might do it. We'd need to ensure the current invariants aren't too slow (< ~0.1s maybe) to cause problems.

cc @gamarin2 @sunnya97 @zmanian thoughts?

[rigelrozanski]

This makes great sense for testnets

[zmanian]

I think it make sense to run nodes on mainnet in this mode.

[zmanian]

@jessysaurusrex FYI

[gamarin2]

As I mentioned to @cwgoes, this should be renamed, as this does not relate to the concept of Circuit breaker. Circuit breaker original goal is to maintain liveness if bugs in specific modules are found by temporarily deactivating these modules. This issue is not about maintaining liveness but halting the chain altogether in case of extreme emergency.

As for the proposal itself, I think all the concerns you put forward are valid. If we do it, we need to hope validators will run their node with this mode as there is no clear incentives to do so.

[alexanderbez]

At the very least, we should benchmark it. If the overhead is relatively minimal, maybe we can enforce it. Otherwise, yes, incentives come into play.

[cwgoes]

At the very least, we should benchmark it. If the overhead is relatively minimal, maybe we can enforce it. Otherwise, yes, incentives come into play.

What if we make it mandatory (put the checks at the end of EndBlock)? Imposing an extra few fractions of a second per block seems a small price to pay for the additional security.

Some invariants will be O(n) in a state that can be altered (accounts, validators) but as long as those cost gas to create I think it's reasonably safe in the short term.

Of course, validators or nodes could elect to remove the code themselves - but I think making the invariants default might be enough to cause wide adoption.