simulation: vanilla simulation tests #71
Conversation
|
@SebastianElvis thank you for the detailed description and the log output, amazing job! I see this: which points here. It looks like Or are you saying this doesn't work because the |
simapp/utils.go
Outdated
| // NOTE: this function is identical to https://github.com/cosmos/cosmos-sdk/blob/v0.45.5/simapp/utils.go. The reason of migrating it here is that it uses the modules' flags initialised in `init()`. Otherwise, if using `sdksimapp.SetupSimulation`, then `sdksimapp.FlagEnabledValue` and `sdksimapp.FlagVerboseValue` will be used, rather than those in `config.go` of our module. | ||
| // The other functions under https://github.com/cosmos/cosmos-sdk/blob/v0.45.5/simapp/utils.go do not access flags and thus can be reused safely. |
There was a problem hiding this comment.
| // NOTE: this function is identical to https://github.com/cosmos/cosmos-sdk/blob/v0.45.5/simapp/utils.go. The reason of migrating it here is that it uses the modules' flags initialised in `init()`. Otherwise, if using `sdksimapp.SetupSimulation`, then `sdksimapp.FlagEnabledValue` and `sdksimapp.FlagVerboseValue` will be used, rather than those in `config.go` of our module. | |
| // The other functions under https://github.com/cosmos/cosmos-sdk/blob/v0.45.5/simapp/utils.go do not access flags and thus can be reused safely. | |
| // NOTE: this function is identical to https://github.com/cosmos/cosmos-sdk/blob/v0.45.5/simapp/utils.go. | |
| // The reason of migrating it here is that it uses the modules' flags initialised in `init()`. Otherwise, | |
| // if using `sdksimapp.SetupSimulation`, then `sdksimapp.FlagEnabledValue` and `sdksimapp.FlagVerboseValue` | |
| // will be used, rather than those in `config.go` of our module. The other functions under | |
| // https://github.com/cosmos/cosmos-sdk/blob/v0.45.5/simapp/utils.go do not access flags and thus can be reused safely. |
|
@SebastianElvis I took a look at the error and turns out that genesis was not setup properly for the |
|
Thanks for the fix!
This issue is because Babylon rejects the existing message types in the staking module and instead requires their wrapped versions. Will investigate and fix (either in this PR or the subsequent ones depending on the complexity and scope). Thanks! |
|
I have fixed the failure caused by unwrapped message types by ruling out simulated messages in the staking module. In addition, I disabled changing the epoch interval on-the-fly in the simulation since this causes errors in the simulation but won't happen in MVP. Now the simulation can run, but still not successfully. I believe the issue is out of this PR's scope -- it will be gone after all modules implement the simulation support. Sample log is attached. |
Partially fixes BM-79
This PR implements the support of all simulation tests specified in
Makefile. Specifically, one can usemake test-sim-<sim_test_name>to run a simulation test, which bootstraps a Babylon SimApp with random parameters and genesis files, triggers certain random operations, and verifies certain correctness properties. Note thatmake testand thus the current CI program does not trigger any simulation test.The current
Makefileprovides the following simulation tests:make test-sim-import-exportmake test-sim-after-importmake test-sim-multi-seed-longmake test-sim-multi-seed-shortmake test-sim-custom-genesis-multi-seedmake test-sim-profilemake test-sim-custom-genesis-fastmake test-sim-nondeterminismmake test-sim-benchmarkmake test-sim-benchmark-invariantsAt the moment, none of them is successful, since we haven't finished the implementation yet. Some sample simulation tests are attached below. This does not mean to be a blocker for our current work -- the integration test phase will happen after we finish implementing all the functionalities. It would be useful for us to find bugs early though.
I also marked @vitsalis as reviewer since Vitalis did the initial migration of
/simappso may have experience on debugging it. Thanks in advance!Results of
make test-sim-import-exportResults of
make test-sim-benchmark