Gaia-13k halt

[cwgoes]

I[2019-03-15|08:11:17.749] Committed state                              module=state height=104438 txs=0 appHash=AE5E0AA871388E6AF38C6543229A59130F57C388B3B480E931182839E09F9450
E[2019-03-15|08:11:23.513] CONSENSUS FAILURE!!!                         module=consensus err="ERROR:\nCodespace: sdk\nCode: 11\nMessage: "19muon,0photino"\n" stack="goroutine 131 [running]:\nruntime/debug.Stack(0xc006a4e178, 0xf8f9a0, 0xc004921cb0)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0xa7\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc000051180, 0x1184458)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:611 +0x57\npanic(0xf8f9a0, 0xc004921cb0)\n\t/usr/local/go/src/runtime/panic.go:513 +0x1b9\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.Hooks.AfterValidatorRemoved(0x1279ce0, 0xc000c530f0, 0xc000b820e0, 0xc000b820e0, 0x1279ce0, 0xc000c53140, 0x1279d20, 0xc000c53150, 0xc000c04620, 0x5, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/hooks.go:50 +0xa16\ngithub.com/cosmos/cosmos-sdk/cmd/gaia/app.StakingHooks.AfterValidatorRemoved(0x1279ce0, 0xc000c530f0, 0xc000b820e0, 0xc000b820e0, 0x1279ce0, 0xc000c53140, 0x1279d20, 0xc000c53150, 0xc000c04620, 0x5, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:358 +0xf2\ngithub.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.AfterValidatorRemoved(0x1279ce0, 0xc000c530c0, 0x1279d20, 0xc000c530d0, 0xc000b820e0, 0x7fec3c8c8878, 0xc000b6e5a0, 0x128b4c0, 0xc000c15440, 0xc000b820e0, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/x/staking/keeper/hooks.go:24 +0xcc\ngithub.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.RemoveValidator(0x1279ce0, 0xc000c530c0, 0x1279d20, 0xc000c530d0, 0xc000b820e0, 0x7fec3c8c8878, 0xc000b6e5a0, 0x128b4c0, 0xc000c15440, 0xc000b820e0, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/x/staking/keeper/validator.go:207 +0x52e\ngithub.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.UnbondAllMatureValidatorQueue(0x1279ce0, 0xc000c530c0, 0x1279d20, 0xc000c530d0, 0xc000b820e0, 0x7fec3c8c8878, 0xc000b6e5a0, 0x128b4c0, 0xc000c15440, 0xc000b820e0, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/x/staking/keeper/validator.go:440 +0x60a\ngithub.com/cosmos/cosmos-sdk/x/staking.EndBlocker(0x1281fe0, 0xc00267d920, 0xc006d08200, 0xd, 0x1279ce0, 0xc000c530c0, 0x1279d20, 0xc000c530d0, 0xc000b820e0, 0x7fec3c8c8878, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/x/staking/handler.go:52 +0x164\ngithub.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc000c16d80, 0x1281fe0, 0xc00267d920, 0xc006d08200, 0xd, 0x197f7, 0x0, 0x0, 0x0, 0x0, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:229 +0x150\ngithub.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0x1281fe0, 0xc00267d920, 0xc006d08200, 0xd, 0x197f7, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:179 +0xc8\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc000b9cc00, 0x197f7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:852 +0x105\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc001fca5a0, 0x197f7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:239 +0xea\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc000c01050, 0x197f7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:77 +0x51\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x1282b60, 0xc000175a20, 0x12879e0, 0xc000c01050, 0xc003bbfe00, 0xc00267cf00, 0x128c2c0, 0xc00000eea0, 0x34b7, 0xc002279600, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:288 +0x701\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc0009d9730, 0xa, 0x0, 0xc000c02f50, 0x6, 0xc000c02f60, 0xa, 0x197f6, 0x34b7, 0xc002279600, ...)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:119 +0x15f\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc000051180, 0x197f7)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1327 +0xa8c\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc000051180, 0x197f7)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1258 +0x451\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit.func1(0xc000051180, 0x0, 0x197f7)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1204 +0x90\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit(0xc000051180, 0x197f7, 0x0)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1235 +0x6b8\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addVote(0xc000051180, 0xc0046f48c0, 0xc005c29410, 0x28, 0xb7efe8, 0xc003c81728, 0xc002f2f7a0)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1660 +0xc1d\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote(0xc000051180, 0xc0046f48c0, 0xc005c29410, 0x28, 0x0, 0x0, 0x15fa053dd41d471b)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1506 +0x59\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc000051180, 0x12763e0, 0xc006bb2448, 0xc005c29410, 0x28)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:694 +0x696\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc000051180, 0x0)\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:642 +0x66f\ncreated by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart\n\t/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:334 +0x13d\n"
E[2019-03-15|08:46:25.409] Failed to reconnect to peer. Giving up       module=p2p addr=5e13a96357d7326b1eff969ef8d96da39896327d@51.38.113.59:26656 elapsed=8h14m37.360824788s
E[2019-03-15|08:53:05.154] Failed to reconnect to peer. Giving up       module=p2p addr=6165404a4b45ab0c2469deda809db89dac5cec01@34.210.245.242:26656 elapsed=8h14m57.730403702s
E[2019-03-15|08:53:25.308] Failed to reconnect to peer. Giving up       module=p2p addr=c24f496b951148697f8a24fd749786075c128f00@35.203.176.214:26656 elapsed=8h14m52.52570007s
E[2019-03-15|08:55:30.468] Failed to reconnect to peer. Beginning exponential backoff module=p2p addr=6165404a4b45ab0c2469deda809db89dac5cec01@34.210.245.242:26656 elapsed=2m25.313268443s
E[2019-03-15|08:55:57.614] Failed to reconnect to peer. Beginning exponential backoff module=p2p addr=c24f496b951148697f8a24fd749786075c128f00@35.203.176.214:26656 elapsed=2m32.306024836s
E[2019-03-15|15:50:03.233] Error on broadcastTxCommit                   module=rpc err="Timed out waiting for tx to be included in a block"

cc @alexanderbez looks related to the truncation issues.

We should determine whether or not this could possibly occur on the Hub mainnet.

[wimel]

goroutine 184732343 [sleep]:
time.Sleep(0x5f5e100)
        /usr/local/go/src/runtime/time.go:105 +0x14f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/mempool.(*MempoolReactor).broadcastTxRoutine(0xc000b72e40, 0x12915e0, 0xc00820c3c0)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/mempool/reactor.go:145 +0x137
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/mempool.(*MempoolReactor).AddPeer
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/mempool/reactor.go:71 +0x53

goroutine 223050384 [chan send, 31 minutes]:
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusReactor).Receive(0xc000143180, 0x405d000000000022, 0x12915e0, 0xc029225140, 0xc055e55000, 0x74, 0x2710)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/reactor.go:305 +0x1eec
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p.createMConnection.func1(0xc01563c222, 0xc055e55000, 0x74, 0x2710)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/peer.go:374 +0x1ba
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn.(*MConnection).recvRoutine(0xc047ba71e0)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn/connection.go:599 +0x8ad
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn.(*MConnection).OnStart
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn/connection.go:210 +0x1cc

goroutine 222789202 [chan send, 193 minutes]:
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusReactor).Receive(0xc000143180, 0x4067e00000000022, 0x12915e0, 0xc02fde46c0, 0xc046daa000, 0xbf, 0x2710)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/reactor.go:305 +0x1eec
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p.createMConnection.func1(0xc0329a0722, 0xc046daa000, 0xbf, 0x2710)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/peer.go:374 +0x1ba
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn.(*MConnection).recvRoutine(0xc02999edc0)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn/connection.go:599 +0x8ad
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn.(*MConnection).OnStart
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn/connection.go:210 +0x1cc

goroutine 222797101 [chan send, 188 minutes]:
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusReactor).Receive(0xc000143180, 0x4067e00000000022, 0x12915e0, 0xc024d15680, 0xc04363b000, 0xbf, 0x2710)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/reactor.go:305 +0x1eec
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p.createMConnection.func1(0xc02b9f8622, 0xc04363b000, 0xbf, 0x2710)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/peer.go:374 +0x1ba
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn.(*MConnection).recvRoutine(0xc037c8c420)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn/connection.go:599 +0x8ad
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn.(*MConnection).OnStart
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn/connection.go:210 +0x1cc

goroutine 222632878 [chan send, 287 minutes]:
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusReactor).Receive(0xc000143180, 0x4066400000000021, 0x12915e0, 0xc0116fe6c0, 0xc036332000, 0xb2, 0x32000)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/reactor.go:279 +0x1724
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p.createMConnection.func1(0xc00979fe21, 0xc036332000, 0xb2, 0x32000)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/peer.go:374 +0x1ba
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn.(*MConnection).recvRoutine(0xc02f640f20)
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn/connection.go:599 +0x8ad
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn.(*MConnection).OnStart
        /home/user/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/conn/connection.go:210 +0x1cc

gaiad version --long
cosmos-sdk: 0.33.0
git commit: 7b4104aced52aa5b59a96c28b5ebeea7877fc4f0
vendor hash: 
build tags: netgo ledger
go version go1.11.5 linux/amd64

• Virtualization: kvm
• Operating System: Ubuntu 16.04.5 LTS
• Kernel: Linux 4.4.0-134-generic
• Architecture: x86-64

I hope this can help.

[jaekwon]

The problem is the 0photino which was returned erroneously from TruncateDecimal.

It appears that the if !coins.IsZero() { check in x/distribution/keeper/hooks.go prevents the bug from manifesting on the cosmos hub, at least.

[alexanderbez]

ref: #3836
ref/dup: #3870

It'll be fixed in that issue's PR. Feel free to close this one.

[cwgoes]

Closing as a duplicate of #3870.