Profiling/improving our tests

[roman-khimov]

We've got a lot of tests which is very good except they take a heck lot of time to run. There were some optimizations already done previously (most notably in #2001), but it's time to make another round. First of all, the most resource-demanding tests need to be identified and then we'll think on whether and how they can be improved.

[fyrchik]

Packages where tests take more than 5 seconds (on my machine):

1. compiler compiler: optimize tests #2382
2. cli -- the biggest (80 seconds), designed to perform end-to-end testing, runs blockchain, waits for blocks. Not sure if we can do anything about it.
3. wallet -- probably stays as it is, most of the time is spent in crypto routines
4. vm -- TestUT, most of the time spent in execute.
5. core

[roman-khimov]

cli -- the biggest (80 seconds), designed to perform end-to-end testing, runs blockchain, waits for blocks. Not sure if we can do anything about it.

It can be split into subpackages which can then be run in parallel (the effect was seen in #1472 already).

My current list is:

ok      github.com/nspcc-dev/neo-go/cli 86.593s coverage: 80.0% of statements
...
ok      github.com/nspcc-dev/neo-go/pkg/compiler        37.964s coverage: 92.0% of statements
...
ok      github.com/nspcc-dev/neo-go/pkg/crypto/keys     7.564s  coverage: 80.8% of statements
...
ok      github.com/nspcc-dev/neo-go/pkg/services/notary 5.021s  coverage: 93.0% of statements
ok      github.com/nspcc-dev/neo-go/pkg/services/oracle 13.095s coverage: 74.7% of statements
...
ok      github.com/nspcc-dev/neo-go/pkg/services/stateroot      8.760s  coverage: 84.3% of statements
...
ok      github.com/nspcc-dev/neo-go/pkg/vm      16.205s coverage: 82.0% of statements
...
ok      github.com/nspcc-dev/neo-go/pkg/wallet  9.996s  coverage: 79.0% of statements

[roman-khimov]

It can be split into subpackages

Argh.

?       github.com/nspcc-dev/neo-go/cli [no test files]
ok      github.com/nspcc-dev/neo-go/cli/app     0.036s  coverage: 100.0% of statements
ok      github.com/nspcc-dev/neo-go/cli/cmdargs 0.011s  coverage: 60.8% of statements
ok      github.com/nspcc-dev/neo-go/cli/flags   0.009s  coverage: 97.7% of statements
?       github.com/nspcc-dev/neo-go/cli/input   [no test files]
ok      github.com/nspcc-dev/neo-go/cli/options 0.033s  coverage: 50.0% of statements
?       github.com/nspcc-dev/neo-go/cli/paramcontext    [no test files]
ok      github.com/nspcc-dev/neo-go/cli/query   2.155s  coverage: 45.3% of statements
ok      github.com/nspcc-dev/neo-go/cli/server  1.373s  coverage: 67.8% of statements
ok      github.com/nspcc-dev/neo-go/cli/smartcontract   8.819s  coverage: 94.3% of statements
ok      github.com/nspcc-dev/neo-go/cli/util    0.006s  coverage: 10.9% of statements
?       github.com/nspcc-dev/neo-go/cli/vm      [no test files]
ok      github.com/nspcc-dev/neo-go/cli/wallet  72.103s coverage: 88.2% of statements

[roman-khimov]

Block verification can be disabled in most of tests to remove some crypto overhead.

[fyfyrchik]

Btw, I have recently stumbled upon https://github.com/gotestyourself/gotestsum which is quite useful

$ go test -json -count=1 ./pkg/... | gotestsum tool slowest --threshold 1s
github.com/nspcc-dev/neo-go/pkg/services/rpcsrv TestClient_Wait 22.53s
github.com/nspcc-dev/neo-go/pkg/services/rpcsrv TestSubClientWaitWithMissedEvent 15.21s
github.com/nspcc-dev/neo-go/pkg/services/rpcsrv TestSubClientWait 9.4s
...