Consensus failure on gaia-6002 #1496
Comments
|
This may be related: #1427 |
|
This is definitely related: #1470 |
|
The only missing validator in the latest height is the zero voting power validator 747A72E570DD41F95687D3C2DD40128A9DBF798A. |
|
This validator - |
|
Possibly a mismatch in Tendermint & SDK-side validator set tracking - #1470 (comment). |
|
Following up, I believe this is a symptom of the combination of #1519 and #1513 - validator Checking that hypothesis by tracing store states. |
|
At block 582090, validator gaiadebug tx "1gHwYl3rD6U26qMKFHaNOFi6UxrMkOx+K4mpynX7VHKeEhR2jThYulMazJDsfiuJqcp1+1RynhoDTUFYBBMOAwERAAAAAAAAAAAEEQAAAAAAAw1ABB4DAQ8WJN5iIOs8d8HI0lIJO+4fGflR+3MT9KJCStTuu81E32eVTHfiFz2h2ypAbWL3f+v6lgW2RQzpnbD/0rHtT5D4Jeal8OPD/yGNbda7mcyUNSwqn5xuJSNp0bjuHqgcrqEGdvGjl5bAsjcAABkAAAAAAAAA5yEAAAAAAAAABAQE"
{
"type": "8EFE47F0625DE8",
"value": {
"msg": {
"type": "ECCB5CA536EAA0",
"value": {
"delegator_addr": "768D3858BA531ACC90EC7E2B89A9CA75FB54729E",
"validator_addr": "768D3858BA531ACC90EC7E2B89A9CA75FB54729E",
"shares": "MAX"
}
},
"fee": {
"amount": [
{
"denom": "",
"amount": 0
}
],
"gas": 200000
},
"signatures": [
{
"pub_key": {
"type": "AC26791624DE60",
"value": "6zx3wcjSUgk77h8Z+VH7cxP0okJK1O67zUTfZ5VMd+I="
},
"signature": {
"type": "6BF5903DA1DB28",
"value": "bWL3f+v6lgW2RQzpnbD/0rHtT5D4Jeal8OPD/yGNbda7mcyUNSwqn5xuJSNp0bjuHqgcrqEGdvGjl5bAsjcAAA=="
},
"account_number": 231,
"sequence": 4
}
]
}
}At that block, this validator wasn't bonded in the application state - so the app did not instruct Tendermint to remove it (no validator set updates) - but it was in Tendermint's validator set, so it remained there (and kept signing blocks!) until later going offline just before the consensus fault, when we tried to slash the validator, couldn't find it in the app state, and panicked. Why was it in Tendermint's validator set? Continuing to trace store updates, we find: At block 571445, an unrevoke transaction was sent for validator gaiadebug tx "uwHwYl3rD7A8uusKFGMeKzrx+f5Ic9Ibe3TO9WjoNPd9BBMOAwERAAAAAAAAAAAEEQAAAAAABhqABB4DAQ8WJN5iIGok5Vt9nXlgYBnpQxbHGQ236WtH17Cy/KBKoTJI4W49Fz2h2ypAZSbGwBzookjGj//4ncSsrzO6eXrWMpF0QqpGbR3gYkRatijiS5LpKYV7m+sq/Iw5V50ILZ3s3qVhhRXADQ41ChkAAAAAAAABLCEAAAAAAAAABAQE"
{
"type": "8EFE47F0625DE8",
"value": {
"msg": {
"type": "A73B89B03CBAE8",
"value": {
"address": "631E2B3AF1F9FE4873D21B7B74CEF568E834F77D"
}
},
"fee": {
"amount": [
{
"denom": "",
"amount": 0
}
],
"gas": 400000
},
"signatures": [
{
"pub_key": {
"type": "AC26791624DE60",
"value": "aiTlW32deWBgGelDFscZDbfpa0fXsLL8oEqhMkjhbj0="
},
"signature": {
"type": "6BF5903DA1DB28",
"value": "ZSbGwBzookjGj//4ncSsrzO6eXrWMpF0QqpGbR3gYkRatijiS5LpKYV7m+sq/Iw5V50ILZ3s3qVhhRXADQ41Cg=="
},
"account_number": 300,
"sequence": 4
}
]
}
}This resulted in two validator set updates. The first was validator Debugging that transaction, logging bonded/unbonded validators, we see: ID CommitID{[218 151 19 112 168 117 250 210 47 69 183 61 136 96 161 151 201 225 160 109]:B02D0}
LastBlockHeight 721616
Txn: BB01F0625DEB0FB03CBAEB0A14631E2B3AF1F9FE4873D21B7B74CEF568E834F77D04130E030111000000000000000004110000000000061A80041E03010F1624DE62206A24E55B7D9D79606019E94316C7190DB7E96B47D7B0B2FCA04AA13248E16E3D173DA1DB2A406526C6C01CE8A248C68FFFF89DC4ACAF33BA797AD632917442AA466D1DE062445AB628E24B92E929857B9BEB2AFC8C39579D082D9DECDEA5618515C00D0E350A19000000000000012C2100000000000000040404
bonded validator 5DE0434D4E5ACA408BB0823B7243B79B1B251717
bonded validator 747A72E570DD41F95687D3C2DD40128A9DBF798A
unbonded validator: 747A72E570DD41F95687D3C2DD40128A9DBF798A
I[07-05|23:00:41.255] Validator 5DE0434D4E5ACA408BB0823B7243B79B1B251717 unrevoked module=x/stake This validator, stored with zero power - #1513 - was incorrectly set as the cliff validator - #1519 - then rebonded and unbonded through the logic in |
|
makes sense as far as I can tell |
This may be related to #1466 and #1484
Right before this consensus failure, I killed off a bunch of my ex-validators. They haven't been voting or bonded for a long time since I didn't have enough steak (you needed 3/4 steaks due to the issues referenced above)
The text was updated successfully, but these errors were encountered: