Enable coverage reporting for test runners

[m-Peter]

Work towards onflow/developer-grants#132

Complementary PRs:

• Enrich the flow test command with coverage features flow-cli#827
• Improve code coverage functionality cadence#2296

Description

• Enable coverage reporting for test runners
• Include also the test script name, when printing test results for each script.

---

• Targeted PR against master branch
• Linked to Github issue with discussion and accepted design OR link to spec that describes this work
• Code follows the standards mentioned here
• Updated relevant documentation
• Re-reviewed Files changed in the Github PR explorer
• Added appropriate labels

[turbolent]

Nice!

[turbolent]

Depends on onflow/cadence#2296, so need to wait for a release that includes it

[SupunS]

👌

[sideninja]

@m-Peter this CI fails, probably need to update the Cadence.

[m-Peter]

@sideninja Yes, I think the release flow is probably this:

1. Release a new cadence version (to include coverage features)
2. Release a new cadence-tools version (has a dependency on cadence)
3. Release a new flow-cli version (has a dependency on both cadence & cadence-tools

[sideninja]

@turbolent

Release a new cadence version (to include coverage features)

Do you have any ETA on that?

[m-Peter]

I was also thinking of randomizing test execution here, with something like:

functions := make(
    []*ast.FunctionDeclaration,
    len(program.Program.FunctionDeclarations()),
)
copy(functions, program.Program.FunctionDeclarations())
rand.Shuffle(len(functions), func(i, j int) {
    functions[i], functions[j] = functions[j], functions[i]
})

for _, funcDecl := range functions {
    ...
}

What are your thoughts on this?

[sideninja]

I was also thinking of randomizing test execution here, with something like:

functions := make(
    []*ast.FunctionDeclaration,
    len(program.Program.FunctionDeclarations()),
)
copy(functions, program.Program.FunctionDeclarations())
rand.Shuffle(len(functions), func(i, j int) {
    functions[i], functions[j] = functions[j], functions[i]
})

for _, funcDecl := range functions {
    ...
}

What are your thoughts on this?

Hmm I'm not sure how I feel about tests not being deterministic, I can see how you could expose flakiness but at the same time it will be hard to identify the source of it I guess due to different execution.

[turbolent]

@sideninja The plan is to cut a Cadence release today/early this week

[m-Peter]

The code coverage seems to be working quite well, with the unit testing functionality provided by the Cadence testing framework.
However, when I used the integration testing functionality, I realized that deployed contracts are "hidden" behind a proxy (I.Test). I have created a gist to showcase that.

After that, I was toying around with the integration testing functionality, and the Test contract. By taking most of the tests from test_framework_test.go, I was able to calculate its code coverage 😅 We could probably utilize this somehow. It was fun anyway, to learn its API.

[m-Peter]

I was also thinking of randomizing test execution here, with something like:

functions := make(
    []*ast.FunctionDeclaration,
    len(program.Program.FunctionDeclarations()),
)
copy(functions, program.Program.FunctionDeclarations())
rand.Shuffle(len(functions), func(i, j int) {
    functions[i], functions[j] = functions[j], functions[i]
})

for _, funcDecl := range functions {
    ...
}

What are your thoughts on this?

Hmm I'm not sure how I feel about tests not being deterministic, I can see how you could expose flakiness but at the same time it will be hard to identify the source of it I guess due to different execution.

That's fair, it can be obtrusive if it is done automatically. However, in a stateful paradigm, I think it can uncover some edge cases that are hard to manifest otherwise. If the test suite is executed in a randomized order, the random seed will be printed out, and the developer can then provide this seed, in order to reproduce the failures.

[sideninja]

That's fair, it can be obtrusive if it is done automatically. However, in a stateful paradigm, I think it can uncover some edge cases that are hard to manifest otherwise. If the test suite is executed in a randomized order, the random seed will be printed out, and the developer can then provide this seed, in order to reproduce the failures.

Yeah, it has an upside to indicate issues, but to my experience the wish for that is usually left to a developer to either state in the tests as part of the testing framework to randomize the execution (which in this case I don't think it will be possible since I don't believe there's an API for it) or by providing an execution flag, which might be applicable here. I will leave the decision to you, just wanted to raise the concern I saw.

[m-Peter]

Yeah, it has an upside to indicate issues, but to my experience the wish for that is usually left to a developer to either state in the tests as part of the testing framework to randomize the execution (which in this case I don't think it will be possible since I don't believe there's an API for it) or by providing an execution flag, which might be applicable here. I will leave the decision to you, just wanted to raise the concern I saw.

Makes sense 👍 I was also thinking about an execution flag as well, to allow developers to specify a randomized execution order. Let's wait and see what Bastian and Supun think about it also.

[sideninja]

Was #2296 already released? is this still blocked by that?

[m-Peter]

@sideninja It was first included in this release: https://github.com/onflow/cadence/releases/tag/v0.34.0 . Let me see if I can get it to work with that version.

[m-Peter]

@sideninja Bumping only the cadence version, doesn't seem to work. We have to also wait for a release of flow-go that includes cadence >= v0.34.0. I see on master branch that it was updated to v0.36.0 (onflow/flow-go@4cd7028). At least, that's my understanding 😅

[turbolent]

Yes, flow-go has been updated. The Emulator was just updated in onflow/flow-emulator#330.

The Emulator will need a couple more improvements (see PR feedback), then after it has been released, I think @dsainati1 will continue updating the downstream dependencies, including the tools here.

[turbolent]

#86 should unblock this PR

[m-Peter]

I will give it a shot! Thanks for the heads up 🙌

[m-Peter]

Well done @turbolent 👏 Rebasing on master, after #86 was merged, seems to have made all checks successful.