Randomized Simulation wishlist

[ValarDragon]

Feel free to edit this issue

1. Prelaunch
   1. Print out the size of the state tree at the end of simulation, so we have an idea how big the state tree is.
   2. Simple benchmarks
   3. Speed up bank simulation
   4. Make make commands obey SIGTERM
   5. Allow operations to specify future operations
   6. Add simulation for governance slashing (description for how to do it in simulation: Allow operations to specify future operations #2166)
   7. Refactor parameters for Operations into structs (including caching pubkeys)
2. Postlaunch
   1. Autopopulate certain types via reflection
   2. Remove the need for operations to need the private key. (Perhaps a "simulation account" struct, which has address, and a sign method) We still want the ability for these to run txs, even though most things should ideally just be checking the handlers. (For different levels of simulation)
   3. Make it simple to pass different loggers to different tx types / invariants.
   4. Make logs flush to disk instead of staying in RAM
   5. Binary search for invariant violation
   6. Use transition matrix for block size

/cc @cwgoes

[rigelrozanski]

We also want to have simple, slow and dumb versions of modules post launch (for staking maybe prelaunch) which can verify the state of the accounts with for each transaction in the randomized tester

[ValarDragon]

followup from: #1783

• Get SigInt working on make commands
• Further document events
• Have governance parameters randomized within acceptable bounds
• make default randomized testing use varying seeds.
• randomize validator ordering in RequestBeginBlock

[cwgoes]

Further document events

Can you expand on this?

Randomize validator ordering in RequestBeginBlock

This isn't necessary, it's deterministic, I read the Tendermint code again - e.g. https://github.com/tendermint/tendermint/blob/013b9cef642f875634c614019ab13b17570778ad/state/execution.go#L244.

[ValarDragon]

Can you expand on this?

Its not clear what the purpose of events are from just looking through the code. I get that operations can emit events, and we log statistics about them, but thats not clear from just looking at the code imo.

This isn't necessary, it's deterministic, I read the Tendermint code again - e.g. https://github.com/tendermint/tendermint/blob/013b9cef642f875634c614019ab13b17570778ad/state/execution.go#L244.

Cool!

Another thing to add to the wishlist:
Make certain operations able to include operations in a later block. The purpose of this would be like for a governance proposal, predetermine when everyone else is going to vote on it.

[ValarDragon]

We also should find a different way of doing our log messages. We keep on appending strings which ends up being O(N^2) due to string immutability.

[ValarDragon]

Tagging prelaunch as I think the current plan is to use this for determining if we even need to worry about gas (or signatures dominate), and to grind a significant amount of situations so we gain confidence in alot of random errors.

[ValarDragon]

Just confirmed, the reason testing is an order of magnitude slower than benchmarks is because of the log additions.

[alexanderbez]

Can we close this @cwgoes ?

[cwgoes]

Most of these are done; binary search is not, but closing is fine for now.