Random Fuzzer panic: calculated final stake greater than current stake #4367

npinto · 2019-05-17T23:03:14Z

Summary of Bug

Running the following command on v0.34.4:
go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=500 -SimulationGenesis= -SimulationVerbose=false -SimulationCommit=true -SimulationSeed=53755686 -SimulationPeriod=50 -v -timeout 24h -SimulationBlockSize=2000

will produce the error:

Simulating... block 99/500, operation 1600/2001. Panic with err
 calculated final stake for delegator cosmos178egu5aws0meqprajjafnes8lgw7qmkfxkhr5q greater than current stake
        final stake:    794207169414.277894338890513826
        current stake:  794207169414.277894338890513825
goroutine 51 [running]:
runtime/debug.Stack(0x10b9f00, 0xc0000d0008, 0xc0226f57f8)
        /snap/go/3739/src/runtime/debug/stack.go:24 +0x9d
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed.func2(0x10c4000, 0xc00128c020, 0xc001282000, 0xc000aad3b0)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/simulation/simulate.go:127 +0xd5
panic(0xce9260, 0xc0147a9970)
        /snap/go/3739/src/runtime/panic.go:522 +0x1b5
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.calculateDelegationRewards(0x10c3f40, 0xc000aacf40, 0xc000ae2070, 0xc000ae2070, 0x10c3f40, 0xc000aacfc0, 0x10c3f80, 0xc000aacfd0, 0xc000aeeea0, 0x5, ...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/delegation.go:106 +0x7be
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards(0x10c3f40, 0xc000aacf40, 0xc000ae2070, 0xc000ae2070, 0x10c3f40, 0xc000aacfc0, 0x10c3f80, 0xc000aacfd0, 0xc000aeeea0, 0x5, ...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/delegation.go:126 +0x31c
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Hooks.BeforeDelegationSharesModified(0x10c3f40, 0xc000aacf40, 0xc000ae2070, 0xc000ae2070, 0x10c3f40, 0xc000aacfc0, 0x10c3f80, 0xc000aacfd0, 0xc000aeeea0, 0x5, ...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/hooks.go:86 +0x1fa
github.com/cosmos/cosmos-sdk/cmd/gaia/app.StakingHooks.BeforeDelegationSharesModified(...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:394
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.BeforeDelegationSharesModified(...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/keeper/hooks.go:52
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.Delegate(0x10c3f40, 0xc000aacf10, 0x10c3f80, 0xc000aacf20, 0xc000ae2070, 0x7fc5c70ee020, 0xc00000c3c0, 0x10dc680, 0xc0004abc20, 0xc000ae2070, ...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/keeper/delegation.go:465 +0x9a1
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.BeginRedelegation(0x10c3f40, 0xc000aacf10, 0x10c3f80, 0xc000aacf20, 0xc000ae2070, 0x7fc5c70ee020, 0xc00000c3c0, 0x10dc680, 0xc0004abc20, 0xc000ae2070, ...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/keeper/delegation.go:681 +0x52c
github.com/cosmos/cosmos-sdk/x/staking.handleMsgBeginRedelegate(0x10d0660, 0xc014bfad80, 0xc01f119e40, 0xd, 0xc000aef400, 0x14, 0x20, 0xc013ec1660, 0x14, 0x14, ...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/handler.go:262 +0x266
github.com/cosmos/cosmos-sdk/x/staking.NewHandler.func1(0x10d0660, 0xc014bfad80, 0xc01f119e40, 0xd, 0x10d3660, 0xc014929560, 0x0, 0x0, 0x0, 0x0, ...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/handler.go:27 +0x8fb
github.com/cosmos/cosmos-sdk/x/staking/simulation.SimulateMsgBeginRedelegate.func1(0xc00a2546f0, 0xc000487400, 0x10d0660, 0xc002353d40, 0xc01f119e40, 0xc, 0xc0000db800, 0x1f, 0x1f, 0x0, ...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/simulation/msgs.go:214 +0xa71
github.com/cosmos/cosmos-sdk/x/simulation.createBlockSimulator.func1(0xc0004a1d10, 0xc000487400, 0x10d0660, 0xc002353d40, 0xc01f119e40, 0xc, 0xc0000db800, 0x1f, 0x1f, 0x0, ...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/simulation/simulate.go:262 +0x58d
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed(0x10e0ba0, 0xc000487800, 0xc000487400, 0xf8f690, 0x3343f26, 0xc000116000, 0xe, 0xe, 0xc00000fc40, 0x3, ...)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/simulation/simulate.go:168 +0x1481
github.com/cosmos/cosmos-sdk/cmd/gaia/app.TestFullGaiaSimulation(0xc000487800)
        /home/cosmos/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/sim_test.go:361 +0x31e
testing.tRunner(0xc000487800, 0xf8f620)
        /snap/go/3739/src/testing/testing.go:865 +0xc0
created by testing.(*T).Run
        /snap/go/3739/src/testing/testing.go:916 +0x35a

Logs to writing to /home/cosmos/.gaiad/simulations/2019-05-18_00:24:34.log
GoLevelDB Stats
Compactions
 Level |   Tables   |    Size(MB)   |    Time(sec)  |    Read(MB)   |   Write(MB)
-------+------------+---------------+---------------+---------------+---------------
   0   |          0 |       0.00000 |       0.31435 |       0.00000 |       2.77626
   1   |          1 |       0.14697 |       0.21408 |       2.93360 |       0.30432

GoLevelDB cached block size 500720
--- FAIL: TestFullGaiaSimulation (547.35s)
    require.go:765:
                Error Trace:    sim_test.go:369
                Error:          Expected nil, but got: &errors.errorString{s:"Simulation halted due to panic on block 99"}
                Test:           TestFullGaiaSimulation
FAIL

Version

$ git log -1
commit 62fa99e715308dcf53de3bec8bee79e2a6a394bc (HEAD, tag: v0.34.4, origin/release/v0.34.4, origin/rc1/v0.34.4)
Author: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Date:   Sat May 4 09:37:15 2019 -0400

    Merge PR #4266: Release v0.34.4 (RC1)

$ gaiad version --long
cosmos-sdk: 0.34.4
git commit: 62fa99e715308dcf53de3bec8bee79e2a6a394bc
vendor hash: b0ee613acca9a3a572b558d0481ec7baa008f732431392af81db3555c92c8dd7
build tags: netgo ledger
go version go1.12.5 linux/amd64

$ gaiacli version --long
cosmos-sdk: 0.34.4
git commit: 62fa99e715308dcf53de3bec8bee79e2a6a394bc
vendor hash: b0ee613acca9a3a572b558d0481ec7baa008f732431392af81db3555c92c8dd7
build tags: netgo ledger
go version go1.12.5 linux/amd64

For Admin Use

Not duplicate issue
Appropriate labels applied
Appropriate contributors tagged
Contributor assigned/self-assigned

The text was updated successfully, but these errors were encountered:

npinto · 2019-05-17T23:19:12Z

Full log:
issue-4367.log

npinto · 2019-05-18T01:48:14Z

Here is another way to replicate this panic:

go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=500 -SimulationGenesis= -SimulationVerbose=false -SimulationCommit=true -SimulationSeed=99516184 -SimulationPeriod=50 -v -timeout 24h -SimulationBlockSize=2000

Output:

...
Simulating... block 161/500, operation 3050/3428. Panic with err
 calculated final stake for delegator cosmos1z0rq9vej70rqr9a9wt2pdf8jfwddhaalwd9ynw greater than current stake
	final stake:	2506467226.583679254317432190
	current stake:	2448222367.754094039806751512
goroutine 52 [running]:
runtime/debug.Stack(0x10b9f00, 0xc0000d0008, 0xc0001ab7f8)
	/snap/go/3739/src/runtime/debug/stack.go:24 +0x9d
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed.func2(0x10c4000, 0xc001803220, 0xc00181c000, 0xc0000c2a00)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/simulation/simulate.go:127 +0xd5
panic(0xce9260, 0xc00e841280)
	/snap/go/3739/src/runtime/panic.go:522 +0x1b5
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.calculateDelegationRewards(0x10c3f40, 0xc00009fc40, 0xc000115ab0, 0xc000115ab0, 0x10c3f40, 0xc00009fc90, 0x10c3f80, 0xc00009fca0, 0xc00003b760, 0x5, ...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/delegation.go:106 +0x7be
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards(0x10c3f40, 0xc00009fc40, 0xc000115ab0, 0xc000115ab0, 0x10c3f40, 0xc00009fc90, 0x10c3f80, 0xc00009fca0, 0xc00003b760, 0x5, ...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/delegation.go:126 +0x31c
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Hooks.BeforeDelegationSharesModified(0x10c3f40, 0xc00009fc40, 0xc000115ab0, 0xc000115ab0, 0x10c3f40, 0xc00009fc90, 0x10c3f80, 0xc00009fca0, 0xc00003b760, 0x5, ...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/hooks.go:86 +0x1fa
github.com/cosmos/cosmos-sdk/cmd/gaia/app.StakingHooks.BeforeDelegationSharesModified(...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:394
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.BeforeDelegationSharesModified(...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/keeper/hooks.go:52
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.Delegate(0x10c3f40, 0xc00009fc10, 0x10c3f80, 0xc00009fc20, 0xc000115ab0, 0x7f9df69fb3c0, 0xc00000c3c0, 0x10dc680, 0xc000f547e0, 0xc000115ab0, ...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/keeper/delegation.go:465 +0x9a1
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.BeginRedelegation(0x10c3f40, 0xc00009fc10, 0x10c3f80, 0xc00009fc20, 0xc000115ab0, 0x7f9df69fb3c0, 0xc00000c3c0, 0x10dc680, 0xc000f547e0, 0xc000115ab0, ...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/keeper/delegation.go:681 +0x52c
github.com/cosmos/cosmos-sdk/x/staking.handleMsgBeginRedelegate(0x10d0660, 0xc024b2f050, 0xc000a30780, 0xd, 0xc000f5ca80, 0x14, 0x20, 0xc007a0c980, 0x14, 0x14, ...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/handler.go:262 +0x266
github.com/cosmos/cosmos-sdk/x/staking.NewHandler.func1(0x10d0660, 0xc024b2f050, 0xc000a30780, 0xd, 0x10d3660, 0xc00e8646c0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/handler.go:27 +0x8fb
github.com/cosmos/cosmos-sdk/x/staking/simulation.SimulateMsgBeginRedelegate.func1(0xc008f930b0, 0xc00048d800, 0x10d0660, 0xc008e4f980, 0xc000a30780, 0xc, 0xc0000e3900, 0x37, 0x37, 0x0, ...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/staking/simulation/msgs.go:214 +0xa71
github.com/cosmos/cosmos-sdk/x/simulation.createBlockSimulator.func1(0xc000f56960, 0xc00048d800, 0x10d0660, 0xc008e4f980, 0xc000a30780, 0xc, 0xc0000e3900, 0x37, 0x37, 0x0, ...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/simulation/simulate.go:262 +0x58d
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed(0x10e0ba0, 0xc00048d600, 0xc00048d800, 0xf8f690, 0x5ee7f18, 0xc000116380, 0xe, 0xe, 0xc000f427e0, 0x3, ...)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/x/simulation/simulate.go:168 +0x1481
github.com/cosmos/cosmos-sdk/cmd/gaia/app.TestFullGaiaSimulation(0xc00048d600)
	/home/cosmos/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/sim_test.go:361 +0x31e
testing.tRunner(0xc00048d600, 0xf8f620)
	/snap/go/3739/src/testing/testing.go:865 +0xc0
created by testing.(*T).Run
	/snap/go/3739/src/testing/testing.go:916 +0x35a

Logs to writing to /home/cosmos/.gaiad/simulations/2019-05-18_03:01:15.log
GoLevelDB Stats
Compactions
 Level |   Tables   |    Size(MB)   |    Time(sec)  |    Read(MB)   |   Write(MB)
-------+------------+---------------+---------------+---------------+---------------
   0   |          0 |       0.00000 |       0.81820 |       0.00000 |       9.26826
   1   |          1 |       0.69384 |       0.80904 |      13.79072 |       5.21630

GoLevelDB cached block size 2509621
--- FAIL: TestFullGaiaSimulation (1680.88s)
    require.go:765:
        	Error Trace:	sim_test.go:369
        	Error:      	Expected nil, but got: &errors.errorString{s:"Simulation halted due to panic on block 161"}
        	Test:       	TestFullGaiaSimulation
FAIL
FAIL	github.com/cosmos/cosmos-sdk/cmd/gaia/app	1681.184s

Full output log:
99516184.log

Full simulation log:
2019-05-18_03:01:15.log.tar.gz

alexanderbez · 2019-05-18T14:05:45Z

Using the following seed on v0.34.4, I do not get this failure. See here.

npinto · 2019-05-18T22:03:49Z

If your cosmos-sdk version is the same, why is the RNG generating different set of parameters on your system? Could this be due to something else?

Your parameters are different (and there are two seeds that produce this bug on my system: 53755686 and 99516184, see above). There seems to be some non-determinism preventing a run on your system from getting the same simulation parameters which is what matters to reproduce the bug (i.e. same code path).

Note that this is related to our discussion here:
#4239 (comment)

Parameters to reproduce the bug (truncated, see full log above):

Selected randomly generated governance parameters:
	{StartingProposalID:17 Deposits:[] Votes:[] Proposals:[] DepositParams:Deposit Params:
  Min Deposit:        571stake
  Max Deposit Period: 11h7m56s VotingParams:Voting Params:
  Voting Period:      11h7m56s TallyParams:Tally Params:
  Quorum:             0.334000000000000000
  Threshold:          0.500000000000000000
  Veto:               0.334000000000000000}

Your parameters (different so won't reproduce the bug):

Selected randomly generated governance parameters:
    {StartingProposalID:17 Deposits:[] Votes:[] Proposals:[] DepositParams:Deposit Params:
  Min Deposit:        90stake
  Max Deposit Period: 11h7m56s VotingParams:Voting Params:
  Voting Period:      11h7m56s TallyParams:Tally Params:
  Quorum:             0.334000000000000000
  Threshold:          0.500000000000000000
  Veto:               0.334000000000000000}

Again, I believe there needs to be a way to inject the exact same parameter set in the tests, or we will have very little guarantee that things that have been tested in the past are still getting tested now...

alexanderbez · 2019-05-18T22:48:12Z

Not necessarily. This is most likely due to the fact of you actually running different code. Did you checkout the release branch or the tag? Also, maybe double check you are indeed running it on v0.34.4 tag.

npinto · 2019-05-20T22:40:38Z

Using the following diff on tag v0.34.4, we can reproduce both bugs:

$ git diff
diff --git a/cmd/gaia/app/sim_test.go b/cmd/gaia/app/sim_test.go
index 80a268ee..798b7be3 100644
--- a/cmd/gaia/app/sim_test.go
+++ b/cmd/gaia/app/sim_test.go
@@ -165,7 +165,7 @@ func appStateRandomizedFn(r *rand.Rand, accs []simulation.Account, genesisTimest
        govGenesis := gov.GenesisState{
                StartingProposalID: uint64(r.Intn(100)),
                DepositParams: gov.DepositParams{
-                       MinDeposit:       sdk.Coins{sdk.NewInt64Coin(sdk.DefaultBondDenom, int64(r.Intn(1e3)))},
+                       MinDeposit:       sdk.Coins{sdk.NewInt64Coin(sdk.DefaultBondDenom, int64(randIntBetween(r, 1, 1e3)))},
                        MaxDepositPeriod: vp,
                },
                VotingParams: gov.VotingParams{

I believe we should re-open this bug, and re-consider hardcoded parameters sets for testing. What do you think?

Thanks!

alexanderbez · 2019-05-21T12:56:49Z

@npinto is the bug DepositParams.MinDeposit? If so, please feel free to open up a PR fixing the param bounds. Also, yes, it would be nice if we could read in parameters from a file :)

npinto · 2019-05-21T17:40:05Z

is the bug DepositParams.MinDeposit?

No. Please, see above.

You can reproduce the exact set of parameters that reproduce the bug if you backport the bug fix for MinDeposit.

alexanderbez closed this as completed May 18, 2019

npinto mentioned this issue May 21, 2019

Random Fuzzer panic: calculated final stake greater than current stake (on f0b690ba6e2) #4383

Closed

4 tasks

noandrea mentioned this issue Apr 4, 2022

Import/Export simulation fails elesto-dao/elesto#83

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Random Fuzzer panic: calculated final stake greater than current stake #4367

Random Fuzzer panic: calculated final stake greater than current stake #4367

npinto commented May 17, 2019

npinto commented May 17, 2019

npinto commented May 18, 2019

alexanderbez commented May 18, 2019

npinto commented May 18, 2019

alexanderbez commented May 18, 2019

npinto commented May 20, 2019

alexanderbez commented May 21, 2019

npinto commented May 21, 2019

Random Fuzzer panic: calculated final stake greater than current stake #4367

Random Fuzzer panic: calculated final stake greater than current stake #4367

Comments

npinto commented May 17, 2019

Summary of Bug

Version

For Admin Use

npinto commented May 17, 2019

npinto commented May 18, 2019

alexanderbez commented May 18, 2019

npinto commented May 18, 2019

alexanderbez commented May 18, 2019

npinto commented May 20, 2019

alexanderbez commented May 21, 2019

npinto commented May 21, 2019