Invariant Benchmarks

[alexanderbez]

Implement BenchmarkInvariants which benchmarks each respective invariant that is registered with the crisis keeper against a simulated gaia app.

Note: The benchmark results for most invariants is extremely negligible with few blocks (< 1000). However, some invariants, namely distr/can-withdraw and staking/delegator-shares can take relatively significant time (>3s). The benchmark values for these two invariants can greatly differ depending on simulation parameters (e.g. seed, number of blocks and block size).

Perhaps we should also attempt to simulate state that is more representative of what we can expect to see on mainnet (100 validators, significant amount of bonds and redelegations, many accounts, etc...)

e.g.

$ go test -mod=readonly -benchmem -run=^$ github.com/cosmos/cosmos-sdk/cmd/gaia/app \
  -bench ^(BenchmarkInvariants)$ -v -SimulationEnabled=true -SimulationNumBlocks=5000 \
  -SimulationBlockSize=200 -SimulationCommit=true -SimulationSeed=57 -timeout 24h

goos: darwin
goarch: amd64
pkg: github.com/cosmos/cosmos-sdk/cmd/gaia/app
BenchmarkInvariants/bank/nonnegative-outstanding-4         	2000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/distr/nonnegative-outstanding-4        	2000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/distr/can-withdraw-4                   	  300000	      3412 ns/op	     422 B/op	       9 allocs/op
BenchmarkInvariants/distr/reference-count-4                	2000000000	         0.02 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/supply-4                       	2000000000	         0.01 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/nonnegative-power-4            	2000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/positive-delegation-4          	2000000000	         0.01 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/delegator-shares-4             	       1	3277646398 ns/op	279621280 B/op	 8234694 allocs/op
PASS
ok  	github.com/cosmos/cosmos-sdk/cmd/gaia/app	80.637s

$ go test -mod=readonly -benchmem -run=^$ github.com/cosmos/cosmos-sdk/cmd/gaia/app \
  -bench ^(BenchmarkInvariants)$ -v -SimulationEnabled=true -SimulationNumBlocks=100 \
  -SimulationBlockSize=200 -SimulationCommit=true -SimulationSeed=57 -timeout 24h

goos: darwin
goarch: amd64
pkg: github.com/cosmos/cosmos-sdk/cmd/gaia/app
BenchmarkInvariants/bank/nonnegative-outstanding-4         	2000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/distr/nonnegative-outstanding-4        	2000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/distr/can-withdraw-4                   	     200	   5353214 ns/op	  634396 B/op	   14093 allocs/op
BenchmarkInvariants/distr/reference-count-4                	2000000000	         0.02 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/supply-4                       	2000000000	         0.01 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/nonnegative-power-4            	2000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/positive-delegation-4          	2000000000	         0.01 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/delegator-shares-4             	       1	3877705648 ns/op	279622432 B/op	 8234706 allocs/op
PASS
ok  	github.com/cosmos/cosmos-sdk/cmd/gaia/app	71.943s

closes: #3914

cc @ValarDragon @cwgoes @rigelrozanski

---

• Targeted PR against correct branch (see CONTRIBUTING.md)

• Linked to github-issue with discussion and accepted design OR link to spec that describes this work.

• Wrote tests

• Updated relevant documentation (docs/)

• Added a relevant changelog entry: sdkch add [section] [stanza] [message]

• rereviewed Files changed in the github PR explorer

---

For Admin Use:

• Added appropriate labels to PR (ex. wip, ready-for-review, docs)
• Reviewers Assigned
• Squashed all commits, uses message "Merge pull request #XYZ: [title]" (coding standards)

[codecov]

Codecov Report

Merging #4079 into develop will increase coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #4079      +/-   ##
===========================================
+ Coverage    59.94%   59.96%   +0.01%     
===========================================
  Files          211      212       +1     
  Lines        15111    15123      +12     
===========================================
+ Hits          9058     9068      +10     
- Misses        5433     5435       +2     
  Partials       620      620

[alessio]

Job very well done.
My only question is: is this really worth a mention in the changelog? Either way, surely not a blocker.

[cwgoes]

ACK - thanks @alexanderbez

A few questions:

• Do the benchmarks change much with seed choice (instead of 57)?
• Have we tried benchmarking simulate-from-genesis with a mainnet export?

[alexanderbez]

Do the benchmarks change much with seed choice (instead of 57)?

The only benchmarks that really change much between seeds are can-withdraw and delegator-shares. Of those two, only can-withdraw fluctuates greatly. The delegator-shares invariants seems to always be very slow.

Have we tried benchmarking simulate-from-genesis with a mainnet export?

Yes, as of block 377715:

go test -mod=readonly ./cmd/gaia/app -benchmem -bench=BenchmarkInvariants -run=^$ \
        -SimulationEnabled=true -SimulationNumBlocks=1000 -SimulationBlockSize=200 \
        -SimulationCommit=true -SimulationSeed=57 -SimulationGenesis ~/Downloads/cosmoshub_1_export_377715.json \
        -v -timeout 24h -benchtime=20s
Starting the simulation from time Sat Mar 26 06:48:06 UTC 6918, unixtime 156151090086

goos: darwin
goarch: amd64
pkg: github.com/cosmos/cosmos-sdk/cmd/gaia/app
BenchmarkInvariants/bank/nonnegative-outstanding-4         	10000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/distr/nonnegative-outstanding-4        	10000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/distr/can-withdraw-4                   	5000000000	         0.14 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/distr/reference-count-4                	10000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/supply-4                       	10000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/nonnegative-power-4            	10000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/positive-delegation-4          	10000000000	         0.00 ns/op	       0 B/op	       0 allocs/op
BenchmarkInvariants/staking/delegator-shares-4             	2000000000	         2.20 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cosmos/cosmos-sdk/cmd/gaia/app	73.292s

Note: I increased the benchtime to 20s. This yeilded better results as far as I can tell.