Bonding Failure - Cannot create any new validator on testnet as of block height 635221

[7768]

I setup a new node running gaiad 0.19.0-c671181 and synced to the top of the chain as the time of writing this issue. Here are the steps I took to verify:

1. First check the status of the node. At the top of the chain. Nothing unusual.

~$ gaiacli status | jq
{
  "node_info": {
    "id": "e7f1ed270cd63b03798102aa9c9e709e1b1a896b",
    "listen_addr": "209.97.148.164:46656",
    "network": "gaia-6002",
    "version": "0.20.0",
    "channels": "4020212223303800",
    "moniker": "7768T1",
    "other": [
      "amino_version=0.9.9",
      "p2p_version=0.5.0",
      "consensus_version=v1/0.2.2",
      "rpc_version=0.7.0/3",
      "tx_index=on",
      "rpc_addr=tcp://0.0.0.0:46657"
    ]
  },
  "sync_info": {
    "latest_block_hash": "6C591DE4D17D856BD80C8777910C3327FAE815ED",
    "latest_app_hash": "3D62385BDCA5548F2B15A4F4F3ACA5A42A1FA3DD",
    "latest_block_height": 635221,
    "latest_block_time": "2018-06-30T01:23:15.924442861Z",
    "syncing": false
  },
  "validator_info": {
    "address": "27A67412FEBE9B35123616AF272F5EB6B4F195AC",
    "pub_key": {
      "type": "AC26791624DE60",
      "value": "mBm62XiR5xkpxeD5nWUuyyPjLhMZAgk+918T2l6fJHE="
    },
    "voting_power": 0
  }
}

2. Created a staking key on the same host for convenience.

~$ gaiacli keys add stakingkey
Enter a passphrase for your key:
Repeat the passphrase:
NAME:	ADDRESS:						PUBKEY:
stakingkey	cosmosaccaddr1cmuxapang4tv7mcxdcyafj49j9g3pmy3ds573t	cosmosaccpub1zcjduc3qma8kcx3emzfeac5lh5fyrc3g6x26cnwa8pflefqnvrfkk9vtytdsa5lf5g
.....

3. Sent one steak to this address from another node, I verify it's received properly.

~$ gaiacli account cosmosaccaddr1cmuxapang4tv7mcxdcyafj49j9g3pmy3ds573t
{
  "type": "6C54F73C9F2E08",
  "value": {
    "address": "C6F86E87B34556CF6F066E09D4CAA5915110EC91",
    "coins": [
      {
        "denom": "steak",
        "amount": 1
      }
    ],
    "public_key": null,
    "account_number": 311,
    "sequence": 0
  }
}

4. This node is obviously not in the validator set yet. Note the error messages here are hard to understand.

~$ echo $STAKEADDR
cosmosaccaddr1cmuxapang4tv7mcxdcyafj49j9g3pmy3ds573t

~$ gaiacli stake validator $STAKEADDR
panic: UnmarshalBinary cannot decode empty bytes

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/go-amino.(*Codec).MustUnmarshalBinary(0xc42008f4a0, 0x0, 0x0, 0x0, 0xc42c80, 0xc420a2ac80)
.....

5. Now bonding and staking by using create-validator command:

~$ gaiacli stake create-validator   --amount=1steak   --pubkey=$(gaiad tendermint show_validator)   --address-validator=$STAKEADDR   --moniker="7768T1"   --chain-id=gaia-6002   --name=stakingkey
Defaulting to account number: 311
Defaulting to next sequence number: 0
Password to sign with 'stakingkey':
Committed at block 635326. Hash: 60D4A7879E07E7240E1B41C10CF31339C1CB21D1

6. Bonding was successful. Let's take a look at the validator.

~$ gaiacli stake validator $STAKEADDR
Validator
Owner: cosmosaccaddr1cmuxapang4tv7mcxdcyafj49j9g3pmy3ds573t
Validator: cosmosvalpub1zcjduc3qnqvm4ktcj8n3j2w9urue6efwev37xtsnrypqj0hhtufa5h5ly3csnc5tnd
Shares: Status Unbonded,  Amount: 976981131321858133/0
Delegator Shares: -7642474989555298493/-8022441456532717568
Description: {7768T1   }
Bond Height: 635326
Proposer Reward Pool:
...

Something is wrong. The newly staked node shows up as Unbonded!

7. Running the same bonding command again and receive an error message as expected.

~$ gaiacli stake create-validator   --amount=1steak   --pubkey=$(gaiad tendermint show_validator)   --address-validator=$STAKEADDR   --moniker="7768T1"   --chain-id=gaia-6002   --name=stakingkey
Defaulting to account number: 311
Defaulting to next sequence number: 1
Password to sign with 'stakingkey':
ERROR: CheckTx failed: (262345) === ABCI Log ===
Codespace: 4
Code:      201
ABCICode:  262345
Error:     --= Error =--
Message: Validator already exist, cannot re-create validator
Cause: <nil>
T: 0xc9
Msg Traces:
--= /Error =--

=== /ABCI Log ===

8. Let's try unrevoke.

~$  gaiacli stake unrevoke   --chain-id=gaia-6002   --name=stakingkey --gas=550000  $STAKEADDR
Defaulting to account number: 311
Defaulting to next sequence number: 1
Password to sign with 'stakingkey':
ERROR: CheckTx failed: (655561) === ABCI Log ===
Codespace: 10
Code:      201
ABCICode:  655561
Error:     --= Error =--
Message: That address is not associated with any known validator
Cause: <nil>
T: 0xc9
Msg Traces:
--= /Error =--

=== /ABCI Log ===

Error message is in direct conflict with the previous command

This is related to
#1466

[7768]

After some more testing, I can reliably reproduce this error now. With the current state of gaia-6002 (block height 717907) you cannot create a validator if you have less than 3steak. The validator becomes unbonded upon creation, and it remains unbonded regardless of how many unrevoke commands you issue.
It works if you have 3 or more steaks!

[cwgoes]

Thanks for the report. Some of the rational print logic was wrong in gaia-6002 (has since been fixed), so I loaded the state manually to check the amount of shares held:

https://github.com/cosmos/cosmos-sdk/tree/6069f60b703d58f00ebcb7f9369ae2b76c53c52e

Checking height 635326 true {C6F86E87B34556CF6F066E09D4CAA5915110EC91 PubKeyEd25519{9819BAD97891E71929C5E0F99D652ECB23E32E131902093EF75F13DA5E9F2471} false {0 4274512115628408635621926372130852035168836285814166273891511693885761997692129474639915776690098653390950677438626430064348952389989837993457247407886041110394437982224413893360667989358203113643967663235756759838227135254561685969671370099063848592790170522461270174678479902250978592328556767498387157296250917937805580634070752361225110037800911300230365222197867850913161591861551517942803701157433862561613265273199306345004087662015054072456687877731191273028620650098798877772144264427973868176719706754718719458184804529038978618903555530449900841624576790984350868141058103611494712062877633833580523963530241095121149611257145910295006470800489267945240755470543594630150996309299180529545108297997591043098562232371646529594790925800467140345037977365700981694641916818356492606871342411156440983699215344949745351619690820821074215769818449256374799510009998619870531991702189133709095456787836181702456338333008975123985673277390258984522556754278857995594472908529750282030369892823609605130251440014694406265251444767526870575343083559601055458472266996569335240129459294730423224546830826151073080832704116275556651871609196039663699881918102510095496480712816844786198234351566610896240749106868025508402276447978498175238045280231459289821377286846860120789720516441255740602270110157372952174087555152577289130628228915911947418687052200757960823482505122538858696407627972017743962341996550180344495168102359126955292178593308610548066774817101623392649552623286624875055749572898682273984444545379871172434762182717421872228375002899498197371235756751584977183151745483694618057799299349026854265123162300132990842576245236539272564751567432249761531439083327053041418346526081443714412754816963700200402628285163133074921701221379618490803024586242808634385455287115713107830646170508015956219079513154981573687837496690517636993727159445804490069305474595568403171209908109410285317345749428158082148048590535991260534841517126303924224467807218023039922188723792302983631455294511189/4268321907432501502040433425659381447765952454802591741982083812179943067907955876742385112385413710065216073306176720258487205111592363255352415903784824713318221311895223692920607926290758549723217265790266100345575963298543934240089902083202937291129668862416492231636989511802613651656876766630430928105945705534495835592915695934262204853984056910192266251460431298501087049515233556936290567862250856059903871217063702547353360353368949329601590998771554711252779053325084673026086950757810620871996876841459320699639844230966663240896872308800256151457547784954312509542389507994626781732168110486375537665779036764586960662885697186605870920701570164423538939789312000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000} 7115943189230531901778262284821355715017266908151902670314439935603102938154538899497879105050786818256386907857986318673297938648371457994222786551389353971777091041945966554841706375006796282909762275564548175550387635525033483456782746649946079341285432690993697468428047008550926107181610353705716355219532921696234655438908067878974794705665148632886499514830971563310058950402007625734921295350878503315820695047768774247797241562904842456899/7115891255492129538671062766420765087279047328787407454538531247331502077665376597080275472845633418297624383819889990021502800511992049901878904435836730797809066472675792619495936340803851268840038562115560333854694095972181833420201258136328328544445727444303243521216864265810788012805751606719087703561894332418886678297045706374483840332353542191741267242599754013093029137132531806164140247017187207351147894181003571861913600000000000000000 {7768T1   } 635326 0  0/1 0/1 0/1 0/1 0/1}
Validator found: owner C6F86E87B34556CF6F066E09D4CAA5915110EC91, pka 27A67412FEBE9B35123616AF272F5EB6B4F195AC, power 0/1, status 0
Checking height 635325 false { <nil> false {0 0/1} 0/1 {   } 0 0  0/1 0/1 0/1 0/1 0/1}

Looks like share calculation is incorrect. edit: Never mind, read it wrong.

[cwgoes]

The printouts are confusing. Understand that creating a validator doesn't necessarily mean it will be bonded, it just creates a candidate - only the top hundred validator candidates at a given block will be actual bonded validators.

The unrevoke command fails since this validator hasn't signed any blocks yet.

There is a bug here - your validator should have been bonded, since there were less than a hundred validators, but it was not. Debugging with https://github.com/cosmos/cosmos-sdk/tree/d61aa0ad2b25605ab9de85f56f307adb15c67914, we see that your validator wasn't bonded because a cliff validator with more power was - incorrectly - stored.

Closing in favor of #1519.