Code Coverage Support for Flow Emulator

[m-Peter]

Code Coverage Support for Flow Emulator

Grant category

• Developer tools / services
• Open source maintenance

Description

The Flow Emulator is the developer tool for the Flow blockchain which allows the development of Cadence smart contracts, scripts and transactions. It is bundled with the flow-cli tool. Among the supported commands of the flow-cli, there is also a test command, which runs tests written with the Cadence testing framework, while utilizing the Flow Emulator under the hood. I would like to enrich this test command, to also provide coverage support, besides just running tests.

Problem statement

Testing is an essential part of the development workflow, and code coverage is an important metric that can help developers view what percentage of all the possible code paths have been exercised! The Cadence core contributors have added support for code coverage a long time ago, but have not exposed this functionality to developers yet.

Proposed solution

The Flow Emulator provides code coverage functionality, when running tests with the Cadence testing framework, with a simple flow-cli command such as:

flow test --coverage coverage.(xml|json|txt) tests.cdc

A preliminary technical discussion can be found on the original issue in the Wishlist column.

A high-level overview of possible changes to provide the desired functionality:

• The interpreter Environment should be initialized with the proper value for the CoverageReportingEnabled config flag, when we wish to generate a CoverageReport, e.g:

env := runtime.NewBaseInterpreterEnvironment(runtime.Config{
	CoverageReportingEnabled: true,
})

• The test command of the flow-cli tool, will need a new flag, e.g:

type flagsTests struct {
	Coverage    string   `default:"coverage.json" flag:"coverage" info:"Filename where the coverage report will be exported."`
}

• The tests services package, should provide a CoverageReport object, to the TestRunner, depending on the value of the coverage config flag, e.g:

runner := cdcTests.NewTestRunner().
	WithImportResolver(t.importResolver(scriptPath, readerWriter)).
	WithFileResolver(t.fileResolver(scriptPath, readerWriter))

if coverageReportingEnabled {
	runner = runner.WithCoverageReport(runtime.NewCoverageReport())
}

Impact

Developers will be able to measure how well their tests cover their contracts, and gain sufficient confidence before deploying to testnet, and more importantly on mainnet.

Milestones and funding

Milestone	Deliverables	Timeline	Risks	USD proposal
1 - MVP	Implementation with detailed documentation in git repo including usage examples	~ 8 weeks	-	22,500
2 - Adoption	Use the new coverage functionality on some of our existing repositories and get 5+ projects on Flow to using the Cadence testing framework with coverage functionality	~ 8 weeks	-	22,500
3 - Maintenance	Resolve and triage issues, feature requests. Write some official documentation.	3 months	-	15,000

Total Cost: 60,000 USD.

Team

Name	Role	Bio	Contact
Ardit Marku	Back-end Engineer	A seasoned back-end engineer with a focus on problem solving and designing robust software systems. Over 8 years of experience building high-traffic search engines, marketplaces, exchanges and in-house analytics solutions. Led cross-functional teams to deliver both web and mobile apps, participating in all steps of the software development lifecycle. Open source lover and advocate.	markoupetr@gmail.com m_peter

[m-Peter]

@chrisackermann I've updated the Milestones and funding section, with an initial cost estimation. Of course, it can be further discussed, in order to reach an agreement 🙏

[chrisackermann]

Hi @m-Peter! Thanks for updating the details on the proposal - we've accepted the proposal and will be reaching out directly on next steps. Thanks!

[m-Peter]

Hi @chrisackermann, I have revisited the proposal to also include Cadence Testing Framework Improvements 🙏

Additionally, I'm excited to share that we had an excellent collaboration with developers from the Flow team, and the Milestone 1 of code coverage has progressed close enough to its completion 🙌

[chrisackermann]

HI @m-Peter! Glad to hear that the collaboration has been awesome!

Would you mind breaking out the Cadence Test Framework Improvements as its own grant in this case? In reading this now, I think it's better for us to manage these as separate grant proposals. Could you please remove from this one and submit as a separate proposal? Thanks!

[m-Peter]

Hey @chrisackermann, I have separated the grant proposals 🙏 The new one can be found here.

[m-Peter]

👋 @chrisackermann Just wanted to let you know, that as of yesterday, there is a new flow-cli version, which comes with code coverage support (https://github.com/onflow/flow-cli/releases/tag/v0.47.0). I have also updated our repository (https://github.com/m-Peter/flow-code-coverage) with usage examples 🙏

[chrisackermann]

@m-Peter - this is excellent, thanks for this!

[m-Peter]

Hey @chrisackermann , another version of flow-cli was released yesterday, which allows running integration tests as well as managing code coverage through the Emulator 🙏

[m-Peter]

@franklywatson The repository containing usage examples about code coverage has moved here: https://github.com/m-Peter/flow-code-coverage. This is where I will put samples for any new features.