Allow standalone coverlet usage for integration/end-to-end tests using .NET tool driver

[MarcoRossignoli]

contributes #781

This is an attempt to support end-to-end coverage scenario.
In this case at the end of tests all generated reports should be moved to a folder and merged with report tools like report generator.

@jakubch1 I know that the workflow you described is a bit different(I call it "cold coverage") but at the moment this is the best we can do.

FYI @AbhitejJohn @pavelhorak

[petli]

Looks good!

A thought about that the test subject process must be shut down somehow: could this be tricky to achieve in CI environments without some help from the coverlet test process? I'm thinking about CI scripts like this, in sh syntax with a hypothetical new flag:

coverlet "/integrationtest" --target "/application/runner.exe" --terminate-test-on-kill &
coverlet_pid=$!
./my_integration_test_suite.sh
kill -TERM $coverlet_pid
wait $coverlet_pid

Based on what I see in e.g. dotnet/coreclr#4309 this would require adding a ProcessExit() handler in coverlet that calls process.Kill() on the test subject. There would be timing issues in this though, since now there's two ProcessExit() events that must happend quickly, which may make it untenable.

Perhaps the recommendation will have to be that the integration test suite have a way to tell the test subject to shut down.

[MarcoRossignoli]

Perhaps the recommendation will have to be that the integration test suite have a way to tell the test subject to shut down.

Added this in guide https://github.com/coverlet-coverage/coverlet/pull/991/files#diff-0ef674889bda556582dddf36bb4f1f71254f7dcfb27a5aeff2f4e5d379582cb1R75 do you think is not enough?

Another idea could be provide a code snipped to use to deterministically flush coverage with some reflection like , CoverletCoverage.FlushCoverage() and looking for tracker.unload call. I mean if a gracefully shutdown is not possible.

A thought about that the test subject process must be shut down somehow: could this be tricky to achieve in CI environments without some help from the coverlet test process?

I think that a test suite has got a workflow like start server -> do tests -> asserts -> stop server -> stop test...do you think that's possible keep a process in hang at the end?

[petli]

Perhaps the recommendation will have to be that the integration test suite have a way to tell the test subject to shut down.

Added this in guide https://github.com/coverlet-coverage/coverlet/pull/991/files#diff-0ef674889bda556582dddf36bb4f1f71254f7dcfb27a5aeff2f4e5d379582cb1R75 do you think is not enough?

True, it's difficult to say anything more specific than that given how diverse the test subjects can be.

Another idea could be provide a code snipped to use to deterministically flush coverage with some reflection like , CoverletCoverage.FlushCoverage() and looking for tracker.unload call. I mean if a gracefully shutdown is not possible.

That's a really good idea! If an integration test anyway have to rely on adding a special endpoint or method to shut down, it would help to ensure that all the hit files are written out too given the limitations on ProcessExit().

A thought about that the test subject process must be shut down somehow: could this be tricky to achieve in CI environments without some help from the coverlet test process?

I think that a test suite has got a workflow like start server -> do tests -> asserts -> stop server -> stop test...do you think that's possible keep a process in hang at the end?

I was thinking it may be a bit fiddly to find the server process to stop, when it is run from coverlet rather than the CI script. This may be very Unix-flavoured of me though, and it would be solved otherwise in a Windows CI environment.

[MarcoRossignoli]

I was thinking it may be a bit fiddly to find the server process to stop, when it is run from coverlet rather than the CI script.

Understood what you mean...if I should do that I would use a mutex to sync test/server...but it's true...depends on how the tests was thought.

We end always at same point, we should use shared memory in a client/server architecture to avoid all problems 😄

[petli]

We end always at same point, we should use shared memory in a client/server architecture to avoid all problems 😄

Indeed :)

[MarcoRossignoli]

@tonerdo PTAL

[tonerdo]

LGTM! Modulo one comment

[jeremyLJ]

The command below on Linux also has the same issue described in #974

still failed after try try with collectors

coverlet TestColverlet/TestColverlet/bin/Debug/netcoreapp3.1/TestColverlet.dll  --target "dotnet" --targetargs "TestColverlet/TestColverlet/bin/Debug/netcoreapp3.1/TestColverlet.dll" --format cobertura --include-test-assembly --verbosity:detailed

[MarcoRossignoli]

@jeremyLJ can you open a new issue? The reason for this kind of fails are a lot so we need to go deeper on your use case.

[jeremyLJ]

@MarcoRossignoli thanks for the quick feedback.
what I met was the exact same issue as https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#6-code-coverage-returns-nan

sorry for the randomization...

this enhancement is pretty cool~