Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic in staking caught by randomized simulation #2063

Closed
cwgoes opened this issue Aug 16, 2018 · 9 comments
Closed

Panic in staking caught by randomized simulation #2063

cwgoes opened this issue Aug 16, 2018 · 9 comments
Assignees
@alexanderbez @rigelrozanski @cwgoes
Labels
C:x/slashing C:x/staking T:Bug

Comments

@cwgoes
Copy link
Contributor

cwgoes commented Aug 16, 2018 •
edited
Loading

On the latest version (0.24.0-rc1 just merged in) of #1783:

--- FAIL: TestFullGaiaSimulation (206.41s)
panic: cliff validator has not been changed, yet we bonded a new validator [recovered]
	panic: cliff validator has not been changed, yet we bonded a new validator

goroutine 23 [running]:
testing.tRunner.func1(0xc420a6bef0)
	/usr/lib/go/src/testing/testing.go:742 +0x29d
panic(0xcf5240, 0x103fd80)
	/usr/lib/go/src/runtime/panic.go:502 +0x229
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.UpdateBondedValidators(0x1045920, 0xc4200ff750, 0xc420a50c40, 0x1045920, 0xc4200ff730, 0xfd0eb8, 0xc420a50c40, 0x4, 0x104cf40, 0xc420ed90b0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:449 +0xf15
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.UpdateValidator(0x1045920, 0xc4200ff750, 0xc420a50c40, 0x1045920, 0xc4200ff730, 0xfd0eb8, 0xc420a50c40, 0x4, 0x104cf40, 0xc420ed90b0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:248 +0x6e3
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.Slash(0x1045920, 0xc4200ff750, 0xc420a50c40, 0x1045920, 0xc4200ff730, 0xfd0eb8, 0xc420a50c40, 0x4, 0x104cf40, 0xc420ed90b0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/slash.go:105 +0xd69
github.com/cosmos/cosmos-sdk/x/gov.EndBlocker(0x104cf40, 0xc420ed90b0, 0xc42093c700, 0x19c, 0xc420a50c40, 0x1045920, 0xc4200ff790, 0x1045920, 0xc4200ff730, 0xfd0eb8, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/gov/handler.go:149 +0x138d
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc4209d8200, 0x104cf40, 0xc420ed90b0, 0xc42093c700, 0x19c, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:149 +0xcb
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0x104cf40, 0xc420ed90b0, 0xc42093c700, 0x19c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:111 +0xce
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc420a560c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:612 +0x107
github.com/cosmos/cosmos-sdk/x/mock/simulation.SimulateFromSeed(0xc420a6bef0, 0xc420a560c0, 0xfcf428, 0x2a, 0xc42e955dd0, 0xc, 0xc, 0xc420b4bcf8, 0x0, 0x0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/random_simulate_blocks.go:106 +0xc08
github.com/cosmos/cosmos-sdk/cmd/gaia/app.TestFullGaiaSimulation(0xc420a6bef0)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/sim_test.go:111 +0x7e2
testing.tRunner(0xc420a6bef0, 0xfcf400)
	/usr/lib/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
	/usr/lib/go/src/testing/testing.go:824 +0x2e0
FAIL	github.com/cosmos/cosmos-sdk/cmd/gaia/app	206.455s
make: *** [Makefile:139: test_sim_gaia_fast] Error 1

Adding a bit of debugging we see:

Slashing validator with owner C5912EE7A5C714054E9EC11D0F34FE1CF510DFFF

UpdateBondedValidators with validator {C5912EE7A5C714054E9EC11D0F34FE1CF510DFFF PubKeyEd25519{0B4DA6E9CEB262ED32F783EB7A17000B28C63B06D7770BF4FDE7C070EC5E4DB9} false 2 13681/100 141/1 {sSRzxflZXs AkNrcbEnHS reZszxZhfj RcqRqrzeDx} 0 15  0/1 0/1 0/1 0/1 0/1}
Deciding to bond: {2D8812BBF99CF8371C5C9D40ED5679F4E0642E6C PubKeySecp256k1{02B5A6188EB73BC5BBAE0B9DCFD72EBD858FEA7915E8FA1D560FCA3344C2DFDBA8} false 0 97/1 97/1 {Lkwtrpwjdf NhnsLFGnir RMDkVKzXpE DrNyZzUVaA} 36 14  0/1 0/1 0/1 0/1 0/1}
Validator: {0312AAE87387BB31CCB9CFAE79041237E99028F3 PubKeyEd25519{7D61971E753F954125E2EE6C3810D73E18F8867C6D098EFFAC714426C1CCE517} false 2 97/1 97/1 {EWZQfPNLaM uxaQPPpvuU eoiAdeywfH ClBwfHyctx} 66 3  0/1 0/1 0/1 0/1 0/1}
Old cliff validator: 0312AAE87387BB31CCB9CFAE79041237E99028F3
Bonded validators count: 100 of max 100

The panic is accurate, we should never hit a state where we bond a new validator but don't change the cliff validator - some earlier prior state transition must be incorrect.
@cwgoes cwgoes self-assigned this Aug 16, 2018
@cwgoes cwgoes mentioned this issue Aug 16, 2018
Merged
@alexanderbez
Copy link
Contributor

alexanderbez commented Aug 16, 2018 •
edited
Loading

@cwgoes, another stack trace (note, I limited to number of blocks to 20 as it took many attempts to get this).

--- FAIL: TestFullGaiaSimulation (3.07s)
panic: already decided to bond a validator, can't bond another! [recovered]
        panic: already decided to bond a validator, can't bond another!

goroutine 53 [running]:
testing.tRunner.func1(0xc42099d0e0)
        /usr/local/go/src/testing/testing.go:742 +0x29d
panic(0x18f44a0, 0x1c3f6d0)
        /usr/local/go/src/runtime/panic.go:502 +0x229
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.UpdateBondedValidators(0x1c453c0, 0xc4204cf440, 0xc4209b7dc0, 0x1c453c0, 0xc4204cf420, 0x1bd0820, 0xc4209b7dc0, 0x4, 0x1c4c9c0, 0xc420f1b500, ...)
        /Users/aleksbez/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:427 +0x102e
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.UpdateValidator(0x1c453c0, 0xc4204cf440, 0xc4209b7dc0, 0x1c453c0, 0xc4204cf420, 0x1bd0820, 0xc4209b7dc0, 0x4, 0x1c4c9c0, 0xc420f1b500, ...)
        /Users/aleksbez/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:248 +0x6e3
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.Delegate(0x1c453c0, 0xc4204cf440, 0xc4209b7dc0, 0x1c453c0, 0xc4204cf420, 0x1bd0820, 0xc4209b7dc0, 0x4, 0x1c4c9c0, 0xc420f1b500, ...)
        /Users/aleksbez/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/delegation.go:250 +0x50c
github.com/cosmos/cosmos-sdk/x/stake.handleMsgDelegate(0x1c4c9c0, 0xc420f1b500, 0xc420d69b00, 0x9, 0xc420efe400, 0x14, 0x20, 0xc420efe2a0, 0x14, 0x20, ...)
        /Users/aleksbez/go/src/github.com/cosmos/cosmos-sdk/x/stake/handler.go:142 +0x39c
github.com/cosmos/cosmos-sdk/x/stake.NewHandler.func1(0x1c4c9c0, 0xc420f1b500, 0xc420d69b00, 0x9, 0x1c4e080, 0xc4209a6a50, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/aleksbez/go/src/github.com/cosmos/cosmos-sdk/x/stake/handler.go:22 +0x5b8
github.com/cosmos/cosmos-sdk/x/stake/simulation.SimulateMsgDelegate.func1(0xc42099d0e0, 0xc42099f770, 0xc4204d2180, 0x1c4c9c0, 0xc420e5e660, 0xc420d69b00, 0x8, 0xc420762000, 0xfa, 0xfa, ...)
        /Users/aleksbez/go/src/github.com/cosmos/cosmos-sdk/x/stake/simulation/msgs.go:107 +0x6ac
github.com/cosmos/cosmos-sdk/x/mock/simulation.SimulateFromSeed(0xc42099d0e0, 0xc4204d2180, 0x1bcedb0, 0x2a, 0xc420c07dd0, 0xc, 0xc, 0xc420e3bcf8, 0x0, 0x0, ...)
        /Users/aleksbez/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/random_simulate_blocks.go:93 +0x90d
github.com/cosmos/cosmos-sdk/cmd/gaia/app.TestFullGaiaSimulation(0xc42099d0e0)
        /Users/aleksbez/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/sim_test.go:111 +0x7e2
testing.tRunner(0xc42099d0e0, 0x1bced88)
        /usr/local/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:824 +0x2e0
FAIL    github.com/cosmos/cosmos-sdk/cmd/gaia/app       3.115s
make: *** [test_sim_gaia_fast] Error 1

Edit: Looks like we have two un-bonded validators in the ValidatorsByPowerIndexKey store that are not revoked. I'll keep debugging...

Edit 2: Looks like in each run when this happens, the affectedValidator happens to be un-bonded and the cliff is another un-bonded validator with power of zero.

@cwgoes
Copy link
Contributor Author

cwgoes commented Aug 16, 2018

Note: this has been merged into the release branch - #1991 - please PR any temporary debugging or attempted fixes against that branch now.

@cwgoes
Copy link
Contributor Author

cwgoes commented Aug 16, 2018 •
edited
Loading

Now that we've fixed the nondeterminism, I reliably get:

Simulating... block 33/1000, operation 5800.UpdateBondedValidators with validator {995F7AF87899ACD2C6E1F692DDE25D9574A1FF39 PubKeySecp256k1{021E67053C485F661FBA2FB6A2B45CAF868FE76C4C51A2F16A020241F8E9D65A79} false 0 122/1 122/1 {axBfsgGaMj   } 33 7  0/1 0/1 0/1 0/1 0/1}
Deciding to bond: {995F7AF87899ACD2C6E1F692DDE25D9574A1FF39 PubKeySecp256k1{021E67053C485F661FBA2FB6A2B45CAF868FE76C4C51A2F16A020241F8E9D65A79} false 0 122/1 122/1 {axBfsgGaMj   } 33 7  0/1 0/1 0/1 0/1 0/1}
Validator: {0E614B95CAC157416D8394DE3C100C86DF0E9E17 PubKeyEd25519{629B6FC6421FAA10AB43F5E009E69659B7EBBDA51E1765CC2D5BA5CB8D82192B} false 2 77/1 77/1 {IYcnIEEPxf mjAsaeOUQF TwQRMgVOEO wUjfutWkkJ} 2 2  0/1 0/1 0/1 0/1 0/1}
Old cliff validator: 0E614B95CAC157416D8394DE3C100C86DF0E9E17
Bonded validators count: 100 of max 100
--- FAIL: TestFullGaiaSimulation (7.71s)
panic: cliff validator has not been changed, yet we bonded a new validator [recovered]
	panic: cliff validator has not been changed, yet we bonded a new validator

goroutine 21 [running]:
testing.tRunner.func1(0xc420a79ef0)
	/usr/lib/go/src/testing/testing.go:742 +0x29d
panic(0xcf5280, 0x103fec0)
	/usr/lib/go/src/runtime/panic.go:502 +0x229
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.UpdateBondedValidators(0x1045a60, 0xc420a6ef50, 0xc420a60c40, 0x1045a60, 0xc420a6ef30, 0xfd0f20, 0xc420a60c40, 0x4, 0x104d080, 0xc420e8e1e0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:458 +0x1033
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.UpdateValidator(0x1045a60, 0xc420a6ef50, 0xc420a60c40, 0x1045a60, 0xc420a6ef30, 0xfd0f20, 0xc420a60c40, 0x4, 0x104d080, 0xc420e8e1e0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:248 +0x6e3
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.Delegate(0x1045a60, 0xc420a6ef50, 0xc420a60c40, 0x1045a60, 0xc420a6ef30, 0xfd0f20, 0xc420a60c40, 0x4, 0x104d080, 0xc420e8e1e0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/delegation.go:250 +0x50c
github.com/cosmos/cosmos-sdk/x/stake.handleMsgCreateValidator(0x104d080, 0xc420e8e1e0, 0xc420743d40, 0x9, 0xc420baf660, 0xa, 0x0, 0x0, 0x0, 0x0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/handler.go:85 +0x530
github.com/cosmos/cosmos-sdk/x/stake.NewHandler.func1(0x104d080, 0xc420e8e1e0, 0xc420743d40, 0x9, 0x104e700, 0xc420ab6be0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/handler.go:18 +0x940
github.com/cosmos/cosmos-sdk/x/stake/simulation.SimulateMsgCreateValidator.func1(0xc420a79ef0, 0xc420ad33b0, 0xc420a660c0, 0x104d080, 0xc4208ab560, 0xc420743d40, 0x8, 0xc420ad8000, 0xfa, 0xfa, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/simulation/msgs.go:46 +0x6c8
github.com/cosmos/cosmos-sdk/x/mock/simulation.SimulateFromSeed(0xc420a79ef0, 0xc420a660c0, 0xfcf490, 0x2a, 0xc420031260, 0xc, 0xc, 0xc420079f40, 0x0, 0x0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/random_simulate_blocks.go:94 +0x90d
github.com/cosmos/cosmos-sdk/cmd/gaia/app.TestFullGaiaSimulation(0xc420a79ef0)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/sim_test.go:133 +0x28c
testing.tRunner(0xc420a79ef0, 0xfcf468)
	/usr/lib/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
	/usr/lib/go/src/testing/testing.go:824 +0x2e0
FAIL	github.com/cosmos/cosmos-sdk/cmd/gaia/app	7.728s
make: *** [Makefile:143: test_sim_gaia_fast] Error 1

@cwgoes
Copy link
Contributor Author

cwgoes commented Aug 16, 2018

Caught one! #2070

Now failing on (using #2068):

Deciding to bond: {E66D218A7DD54DA493D5065E8CC6376BCFEAD4C0 PubKeyEd25519{3DCD79983D00C7346F4FE1EABD6FCE6920EB422532A043FD5A471912B52C237B} false 0 86/1 86/1 {kXIRhVgWxK AeqALvweff vjneWNqPHm XSyFTvXsPb} 55 13  0/1 0/1 0/1 0/1 0/1}
Validator: {A617CEB2347C6CED3FDFED2C0670E756F6066BB2 PubKeySecp256k1{0228F76B8CF6B76B7C6C82A4F01AD59810C50E4D50FCB92ECAD1D645D4DCFAA1A3} false 0 85/1 85/1 {KQFQtTkXAd qdaguORCuZ qPEgLbANwQ PdJePIMLPC} 12 8  0/1 0/1 0/1 0/1 0/1}
--- FAIL: TestFullGaiaSimulation (21.55s)
panic: already decided to bond a validator, can't bond another! [recovered]
	panic: already decided to bond a validator, can't bond another!

goroutine 36 [running]:
testing.tRunner.func1(0xc420761c20)
	/usr/lib/go/src/testing/testing.go:742 +0x29d
panic(0xcf6280, 0x1041130)
	/usr/lib/go/src/runtime/panic.go:502 +0x229
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.UpdateBondedValidators(0x1046ce0, 0xc420493c70, 0xc42078b110, 0x1046ce0, 0xc420493c50, 0xfd2000, 0xc42078b110, 0x4, 0x104e300, 0xc420e3cdb0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:447 +0x1228
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.UpdateValidator(0x1046ce0, 0xc420493c70, 0xc42078b110, 0x1046ce0, 0xc420493c50, 0xfd2000, 0xc42078b110, 0x4, 0x104e300, 0xc420e3cdb0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:248 +0x6e3
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.Delegate(0x1046ce0, 0xc420493c70, 0xc42078b110, 0x1046ce0, 0xc420493c50, 0xfd2000, 0xc42078b110, 0x4, 0x104e300, 0xc420e3cdb0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/delegation.go:250 +0x50c
github.com/cosmos/cosmos-sdk/x/stake.handleMsgDelegate(0x104e300, 0xc420e3cdb0, 0xc420f2be00, 0x9, 0xc4214c2a60, 0x14, 0x20, 0xc4214c2a00, 0x14, 0x20, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/handler.go:142 +0x39c
github.com/cosmos/cosmos-sdk/x/stake.NewHandler.func1(0x104e300, 0xc420e3cdb0, 0xc420f2be00, 0x9, 0x104f9c0, 0xc4212f8370, 0x0, 0x0, 0x0, 0x0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/handler.go:22 +0x5b8
github.com/cosmos/cosmos-sdk/x/stake/simulation.SimulateMsgDelegate.func1(0xc420761c20, 0xc420773620, 0xc4200100c0, 0x104e300, 0xc420b8e2d0, 0xc420f2be00, 0x8, 0xc420a5f000, 0xfa, 0xfa, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/stake/simulation/msgs.go:107 +0x6ac
github.com/cosmos/cosmos-sdk/x/mock/simulation.SimulateFromSeed(0xc420761c20, 0xc4200100c0, 0xfd0570, 0x2a, 0xc420031140, 0xc, 0xc, 0xc420079f40, 0x0, 0x0, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/random_simulate_blocks.go:94 +0x90d
github.com/cosmos/cosmos-sdk/cmd/gaia/app.TestFullGaiaSimulation(0xc420761c20)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/sim_test.go:133 +0x28c
testing.tRunner(0xc420761c20, 0xfd0548)
	/usr/lib/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
	/usr/lib/go/src/testing/testing.go:824 +0x2e0
FAIL	github.com/cosmos/cosmos-sdk/cmd/gaia/app	21.581s

@rigelrozanski
Copy link
Contributor

awesome work

@cwgoes
Copy link
Contributor Author

cwgoes commented Aug 16, 2018

Caught another! #2071

@cwgoes
Copy link
Contributor Author

cwgoes commented Aug 17, 2018

OK I think this is essentially catching #1676 now:

Validator: {05306235BDA68FBED2E2C75CE4F000ACC5F6ED55 PubKeySecp256k1{03AF771809DC83374BAF9411F61790E09AF4DE939E40CD64401773D33F73D9239B} false 2 102/1 102/1 {zOENEgNRRh ADWZIDsrLT ulUYlnNWet GsYpbxlLpE} 2 38  0/1 0/1 0/1 0/1 0/1}
Validator: {CD726469C08299508995BF7F8AAF814BF4AA192D PubKeySecp256k1{028A734A8E2CE426A312F36BC987EA23C86944828D705106C19A1BE781366ADBCD} false 2 102/1 102/1 {kmUTDGFofU   } 42 3  0/1 0/1 0/1 0/1 0/1}
Validator: {1F289B01950D44AA59118E5D97ACC04E1A3CA727 PubKeySecp256k1{03B14668BE7C68B6A410717647B9F94A24E0536E00B581EE6F34E9A0FE7D71D464} false 2 101/1 101/1 {   } 0 16  0/1 0/1 0/1 0/1 0/1}
Validator: {286E3DA9636F33E91D8C744BCD1C2797A10FBD2D PubKeySecp256k1{0310A6193C5A9CF85E7B1901D07EF04B0EC0708D7C9DC582E70C1B3A2ED7CD68AD} false 2 100/1 100/1 {WYbYOhjJeU LCNumXFrAN TXOjHsyKNS ZVeQtzhMEU} 0 17  0/1 0/1 0/1 0/1 0/1}
Validator: {EF0DC8F745C5988DA2B876211D6F7063602D3DC1 PubKeyEd25519{FFBE1E948897E71855F6597C450C71CFBF918A8B9F072A793F3E33111B07F9A1} false 2 100/1 100/1 {SyUMVxTKZA IpLRQNLKQv zYIyAHlQrD tLLivzBCss} 0 25  0/1 0/1 0/1 0/1 0/1}
Validator: {47FCA00D5C08B6D73F80DC91C3852CC88C6DE708 PubKeyEd25519{71C144EDA19F7AFBA0A062B0A11114DF245B865B6F5EBCECADDDA405352A8A72} false 2 100/1 100/1 {WwIgtbPufm OzVnLEzlHg WYWOEFSOgh yKgJxmbhbQ} 0 42  0/1 0/1 0/1 0/1 0/1}
Validator: {D3F27F3D7CC915CAE013BA6A995B2EBE98EFA085 PubKeySecp256k1{028DD9F397AC1000BE7EA301C837522A2576BC47FA9D0AC5DA81F5F4DBC11A6FE9} false 2 100/1 100/1 {jgBAVKyMgU awITaMXKRY rqGaOQsuae kLgyqsMtQB} 0 44  0/1 0/1 0/1 0/1 0/1}
Validator: {C994CFDC11D4D4155FA6027E54EBA4CFC792EB30 PubKeySecp256k1{03C69D5BE445914A436F852ADB94B49FA725127CD002D19CE8B46C47EA86050C8D} false 2 100/1 100/1 {ighhxodozr   } 27 7  0/1 0/1 0/1 0/1 0/1}
Validator: {A3BFA54C4B11A406509DB9178AC15388038753D3 PubKeyEd25519{C11C288C24A0D7B2E3FBA815098CEBE4D60C147EE18A08C1B49ADACA91CC6E5C} false 2 99/1 99/1 {PqgqOZWuEF dJobSPcXVC jptSHJHmBz TuWEjjLWZj} 2 14  0/1 0/1 0/1 0/1 0/1}
Validator: {243593D4849930BC72A3AAA396273D3C0B908838 PubKeySecp256k1{0372D7268DDFECAA0062BD963E19F341781781A226D6A2AE85315A3AD09D0F48F4} false 2 97/1 97/1 {gHPESxgMXZ ZakxadHLdP SvvTvLkvUb tvfWLDDGKz} 17 13  0/1 0/1 0/1 0/1 0/1}
Validator: {1229AAD3ABFF23405ED64CEFC98393180F711E23 PubKeyEd25519{8E6FC8A7CB41ACD56DD417D9FA3C2C00610C4BD7BA46B87625F7FC97C1D09AB8} false 2 95/1 95/1 {nyqXeFdhcf hCAGZeLSWX wxluhroHqj TOtZMOorCh} 33 22  0/1 0/1 0/1 0/1 0/1}
Validator: {8E36BB4ABE80BD03417DB053BDC6C9A84297CEA4 PubKeySecp256k1{035146E8E3D7D9B8FC3A790D3D0134CE8AE23930E866CB3BA5363D2980715449AB} false 2 94/1 94/1 {CHncEKQDCN jMHusZmWOy aVSvEEVAwI tglCdPDZfC} 2 35  0/1 0/1 0/1 0/1 0/1}
Validator: {B70F1A7FE92C3183A1E970EA4F6669881D544524 PubKeySecp256k1{03760B462CA7AFCAC9BDE4317896BB2A3464EB0E5363F4BA2F938B13F9AA4F1345} false 2 94/1 94/1 {NeEsYXECeS CCjzKUJfsw lYYyiiNHQQ ozDLgCxEnb} 33 10  0/1 0/1 0/1 0/1 0/1}
Validator: {A1B337EEC237C6143757EA0BEB91061FF0D403D6 PubKeySecp256k1{02E7854A019663E523780AEB5FC6BACC0BE63054CADB81B2853468D6CC22C1E580} false 2 89/1 89/1 {teDSYmYOgH fcUBWWDkaA NOKCZuZQPp nSNZIibWej} 30 0  0/1 0/1 0/1 0/1 0/1}
Validator: {B8374801DD77FA4D87D70814E162F150FBC86223 PubKeyEd25519{F7468869F233A847FA57F77B04FD34707C775DFDD96DAFD5A57DF275CF2D63FD} false 2 86/1 86/1 {yKaHIjBCFh CYDtwcpcNM mWIWAsrWpK QuSyWtjNBV} 30 9  0/1 0/1 0/1 0/1 0/1}
Validator: {A617CEB2347C6CED3FDFED2C0670E756F6066BB2 PubKeySecp256k1{0228F76B8CF6B76B7C6C82A4F01AD59810C50E4D50FCB92ECAD1D645D4DCFAA1A3} false 2 85/1 85/1 {KQFQtTkXAd qdaguORCuZ qPEgLbANwQ PdJePIMLPC} 12 8  0/1 0/1 0/1 0/1 0/1}
Validator: {72B106636C1B85D0B5818B5CB7184C99742B6F0B PubKeySecp256k1{03BA106E258391F58B200EA2A103DDC8F6E7C078446E7B38DEF3B15DAC8B112596} false 0 83/1 83/1 {thcMnwrXzu sPfJeVvHIW lhqarjvZsY ZDXKvHBkvQ} 33 16  0/1 0/1 0/1 0/1 0/1}
Deciding to bond: {72B106636C1B85D0B5818B5CB7184C99742B6F0B PubKeySecp256k1{03BA106E258391F58B200EA2A103DDC8F6E7C078446E7B38DEF3B15DAC8B112596} false 0 83/1 83/1 {thcMnwrXzu sPfJeVvHIW lhqarjvZsY ZDXKvHBkvQ} 33 16  0/1 0/1 0/1 0/1 0/1}
Setting cliff validator to 72B106636C1B85D0B5818B5CB7184C99742B6F0B
unbonding validator: 2E7893C7CB4E3E21B40110E15E20338D04078773
--- FAIL: TestFullGaiaSimulation (1066.13s)
  assertions.go:247: ^M                          ^M Error Trace:  invariants.go:55
    ^M                ^M      invariants.go:21
    ^M                ^M      sim_test.go:110
    ^M                ^M      random_simulate_blocks.go:210
    ^M                ^M      random_simulate_blocks.go:99
    ^M                ^M      sim_test.go:133
    ^M  Error:        Should be true
    ^M  Test:         TestFullGaiaSimulation
    ^M  Messages:     expected loose tokens to equal total steak held by accounts - pool.LooseTokens: 6350, sum of account tokens: 6276
    ^M                log: Starting SimulateFromSeed with randomness created with seed 42
    ^M                BeginBlock
    ^M                EndBlock
    ^M                event - beginblock/signing/signed
    ^M                event - beginblock/signing/signed
    ^M                event - beginblock/signing/missed
    ^M                event - beginblock/signing/signed
    ^M                event - beginblock/signing/signed

Once a validator is unbonded, the tokens are added back to the pool but don't go to any accounts or to any unbonding delegations, and they cannot be recovered (unless the validator is later luckily rebonded).

@cwgoes cwgoes mentioned this issue Aug 17, 2018
Merged
3 tasks
@ebuchman
Copy link
Member

Fixed ?

@cwgoes
Copy link
Contributor Author

cwgoes commented Aug 20, 2018

Can be closed in favor of #1676.

@cwgoes cwgoes closed this as completed Aug 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexanderbez alexanderbez

@rigelrozanski rigelrozanski

@cwgoes cwgoes

Labels
C:x/slashing C:x/staking T:Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ebuchman @alexanderbez @rigelrozanski @cwgoes