gaiad ran until synced, now won't start

[proofoftom]

Ubuntu 18.04.1 LTS
Go 1.11.2

I was running the node in the background for a few days to sync, came back and everything was still chugging away and it appeared to no longer be syncing. After stopping the node it would no longer start though. I'm not sure what relevant information to provide so here's the output from trying to start the node, let me know what else I can provide:

cryptotom@cryptotom-MS-7693:~$ gaiad start
I[12116-11-12|22:27:51.181] Starting ABCI with Tendermint                module=main 
I[12116-11-12|22:27:51.693] Starting multiAppConn                        module=proxy impl=multiAppConn
I[12116-11-12|22:27:51.693] Starting localClient                         module=abci-client connection=query impl=localClient
I[12116-11-12|22:27:51.693] Starting localClient                         module=abci-client connection=mempool impl=localClient
I[12116-11-12|22:27:51.693] Starting localClient                         module=abci-client connection=consensus impl=localClient
I[12116-11-12|22:27:51.693] ABCI Handshake App Info                      module=consensus height=33277 hash=365A634BD12A86E0E8DABFADAF311623C887EA59A186EB6C5CE1D30D0AA37B7E software-version= protocol-version=0
I[12116-11-12|22:27:51.693] ABCI Replay Blocks                           module=consensus appHeight=33277 storeHeight=33278 stateHeight=33277
I[12116-11-12|22:27:51.693] Replay last block using real app             module=consensus 
I[12116-11-12|22:27:51.718] Absent validator 0933955E48196F0759A546A8EB9AA5356B1E342F at height 33278, 1062 missed, threshold 2500 module=x/slashing 
I[12116-11-12|22:27:51.727] Absent validator 39564FAB61C1686E63DCEF78ECB9DA3448E4EFBB at height 33278, 588 missed, threshold 2500 module=x/slashing 
I[12116-11-12|22:27:51.737] Absent validator 703F066587197E98C5989C0F4F9A90E5C29F1879 at height 33278, 2501 missed, threshold 2500 module=x/slashing 
I[12116-11-12|22:27:51.737] Validator 703F066587197E98C5989C0F4F9A90E5C29F1879 past min height of 31427 and below signed blocks threshold of 2500 module=x/slashing 
panic: ERROR:
Codespace: 1
Code: 10
Message: "47steak < -121932steak"


goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.WithdrawToDelegator(0x11d6ae0, 0xc0001cf700, 0xc00020eb60, 0xc00020eb60, 0x11d6ae0, 0xc0001cf750, 0x11d6b20, 0xc0001cf760, 0xc000b56bc0, 0x5, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/delegation.go:120 +0x34b
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.WithdrawValidatorRewardsAll(0x11d6ae0, 0xc0001cf700, 0xc00020eb60, 0xc00020eb60, 0x11d6ae0, 0xc0001cf750, 0x11d6b20, 0xc0001cf760, 0xc000b56bc0, 0x5, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/validator.go:77 +0x6a6
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.onValidatorModified(0x11d6ae0, 0xc0001cf700, 0xc00020eb60, 0xc00020eb60, 0x11d6ae0, 0xc0001cf750, 0x11d6b20, 0xc0001cf760, 0xc000b56bc0, 0x5, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/hooks.go:28 +0x120
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Hooks.OnValidatorModified(0x11d6ae0, 0xc0001cf700, 0xc00020eb60, 0xc00020eb60, 0x11d6ae0, 0xc0001cf750, 0x11d6b20, 0xc0001cf760, 0xc000b56bc0, 0x5, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/hooks.go:104 +0xd0
github.com/cosmos/cosmos-sdk/cmd/gaia/app.Hooks.OnValidatorModified(0x11d6ae0, 0xc0001cf700, 0xc00020eb60, 0xc00020eb60, 0x11d6ae0, 0xc0001cf750, 0x11d6b20, 0xc0001cf760, 0xc000b56bc0, 0x5, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:345 +0xd0
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.OnValidatorModified(0x11d6ae0, 0xc0001cf6d0, 0x11d6b20, 0xc0001cf6e0, 0xc00020eb60, 0x11e4900, 0xc0001ed6c0, 0x11e6040, 0xc0000beb40, 0xc00020eb60, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/hooks.go:16 +0xa5
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.Slash(0x11d6ae0, 0xc0001cf6d0, 0x11d6b20, 0xc0001cf6e0, 0xc00020eb60, 0x11e4900, 0xc0001ed6c0, 0x11e6040, 0xc0000beb40, 0xc00020eb60, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/slash.go:54 +0x584
github.com/cosmos/cosmos-sdk/x/slashing.Keeper.handleValidatorSignature(0x11d6ae0, 0xc0001cf720, 0xc00020eb60, 0x11e77e0, 0xc00051dae0, 0xc00020eb60, 0x11d6ae0, 0xc0001cf750, 0x11d6b20, 0xc0001cf760, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/x/slashing/keeper.go:156 +0xc0b
github.com/cosmos/cosmos-sdk/x/slashing.BeginBlocker(0x11deb60, 0xc00023b0b0, 0xc00007be40, 0xb, 0xc000b57c60, 0x20, 0x20, 0x7, 0x0, 0x0, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/x/slashing/tick.go:24 +0x2d8
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).BeginBlocker(0xc000186c00, 0x11deb60, 0xc00023b0b0, 0xc00007be40, 0xb, 0xc000b57c60, 0x20, 0x20, 0x7, 0x0, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:193 +0xea
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).BeginBlocker-fm(0x11deb60, 0xc00023b0b0, 0xc00007be40, 0xb, 0xc000b57c60, 0x20, 0x20, 0x7, 0x0, 0x0, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:164 +0xba
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(0xc0008d05a0, 0xc000b57c60, 0x20, 0x20, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:434 +0x230
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(0xc00007c480, 0xc000b57c60, 0x20, 0x20, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:206 +0x9e
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(0xc0001cfed0, 0xc000b57c60, 0x20, 0x20, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:69 +0x6b
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x11df7e0, 0xc0008c32a0, 0x11e4480, 0xc0001cfed0, 0xc0004ae1e0, 0xc000913380, 0x11e8e60, 0xc0000b6740, 0x35d3, 0xc0009be600, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:235 +0x47f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc00010b180, 0x7, 0x0, 0xc0001ab506, 0x6, 0xc0001ab520, 0x9, 0x81fd, 0x35d3, 0xc0009be600, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:96 +0x15f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(0xc0000c8b00, 0x7, 0x0, 0xc0001ab506, 0x6, 0xc0001ab520, 0x9, 0x81fd, 0x35d3, 0xc0009be600, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/replay.go:423 +0x20d
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(0xc0000c8b00, 0x7, 0x0, 0xc0001ab506, 0x6, 0xc0001ab520, 0x9, 0x81fd, 0x35d3, 0xc0009be600, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/replay.go:354 +0x7b4
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc0000c8b00, 0x11e98e0, 0xc00010acb0, 0x2, 0x11df7e0)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/replay.go:252 +0x4c8
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/node.NewNode(0xc000888900, 0x11e0de0, 0xc0008ce140, 0xc0001cfe60, 0x11d3940, 0xc0008c2780, 0xc0001a6150, 0x10f2e88, 0xc0001a6160, 0x11df7e0, ...)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/node/node.go:203 +0x6f2
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc0001ecd80, 0x10f36f0, 0x1d, 0x0, 0x0)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/server/start.go:118 +0x3f1
github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc0000cafc0, 0x1bf3fc0, 0x0, 0x0, 0x0, 0x0)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/server/start.go:40 +0xab
github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra.(*Command).execute(0xc0000cafc0, 0x1bf3fc0, 0x0, 0x0, 0xc0000cafc0, 0x1bf3fc0)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra/command.go:698 +0x47a
github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00025afc0, 0xc0001ecfe0, 0xc000af1e78, 0xc2b2ed)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra/command.go:783 +0x2dc
github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra.(*Command).Execute(0xc00025afc0, 0xc2aba2, 0xc0001cf0a0)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra/command.go:736 +0x2b
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/libs/cli.Executor.Execute(0xc00025afc0, 0x10f3ad8, 0x2, 0xc000b56060)
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/libs/cli/setup.go:89 +0x4e
main.main()
	/home/cryptotom/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/cmd/gaiad/main.go:51 +0x385

[alexanderbez]

This is related to #2772 most likely. I don't think we need a separate issue for this.

[cwgoes]

Closing as a duplicate of #2777.