Coverage collection requires instrumented code in same location until collection is not completed causing service downtime

[rinkal-chomal7]

Hi Steve,

For the collecting coverage we need the dotnet service folder and the folder with the .acv files in the state when we have stopped the service. After which we can run command to collect coverage and it takes 40-50 mins in our service scenario which is resulting into down time of the service.

Can you suggest a way if possible to move this instrumented dotnet service (i.e., __Saved folder) and acv files folder to other place and restart the service for our usage?

[SteveGilham]

First, let me congratulate you on the bravery of running live for testing.

The issue as stated in the title "Coverage collection requires instrumented code in same location until collection is not completed causing service downtime" - presumably a stray "not", or meant as "while ... not" - can be circumvented using the "Instrument now, test later, collect coverage after that" style

1. instrument sending the saved code to a location not under the service directory, and the report (and .acv files) in a third separate location

AltCover -i /path/to/LiveService --inplace -o /path/to/Saved -r /somewhere/else/coverage.xml --save

2. Start service, run tests, stop service, rename /path/to/LiveService to /path/to/Tested, and then /path/to/Saved as /path/to/LiveService. Start the saved service and resume normal operation

3. Collect the data at leisure

AltCover runner --collect -r `/path/to/Tested`

Now as to the length of time taken to process the coverage data being hundreds or thousands of times longer than observed in the largest test cases I have to hand (altcover self-testing while unit tests are run) - this is essentially the same as issue #175

I trust that you have ensured that the coverage instrumentation has excluded by filters any third party assemblies and such which bulk up the output to no good effect. It may also help to use the --single option which records only the first visit to any location.

The question is then how much data are being processed across what size of report. To at least assist me with understanding the scaling here, could you provide the summary data for a run - as an example, this is the summary data for the altcover self-test in the most recent (at time of writing) GitHub actions build -

2024-05-16T09:32:37.5541051Z Getting results...
2024-05-16T09:32:37.5797018Z ... D:\a\altcover\altcover\_Binaries\AltCover.Tests\Debug+AnyCPU\net472\UnitTestWithAltCoverRunner.xml.0.acv (66,267b)
2024-05-16T09:32:37.6068420Z 24,047 visits recorded in 00:00:00.0495828 (484,987 visits/sec)
2024-05-16T09:32:37.6069827Z ... D:\a\altcover\altcover\_Binaries\AltCover.Tests\Debug+AnyCPU\net472\UnitTestWithAltCoverRunner.xml.1.acv (99b)
2024-05-16T09:32:37.6071792Z 14 visits recorded in 00:00:00.0002717 (51,527 visits/sec)
2024-05-16T09:32:37.6072914Z ... D:\a\altcover\altcover\_Binaries\AltCover.Tests\Debug+AnyCPU\net472\UnitTestWithAltCoverRunner.xml.2.acv (99b)
2024-05-16T09:32:37.6074294Z 14 visits recorded in 00:00:00.0001714 (81,680 visits/sec)
2024-05-16T09:32:37.6075399Z ... D:\a\altcover\altcover\_Binaries\AltCover.Tests\Debug+AnyCPU\net472\UnitTestWithAltCoverRunner.xml.3.acv (99b)
2024-05-16T09:32:37.6076628Z 14 visits recorded in 00:00:00.0001786 (78,387 visits/sec)
2024-05-16T09:32:37.6079323Z A total of 24,089 visits recorded
2024-05-16T09:32:38.4285411Z Coverage statistics flushing took 0.81 seconds
2024-05-16T09:32:38.6237659Z Visited Classes 1732 of 1734 (99.88)
2024-05-16T09:32:38.6238429Z Visited Methods 2818 of 2824 (99.79)
2024-05-16T09:32:38.6239000Z Visited Points 18542 of 18570 (99.85)
2024-05-16T09:32:38.6239477Z Visited Branches 3455 of 5908 (58.48)
2024-05-16T09:32:38.6239887Z Maximum CRAP score 28
2024-05-16T09:32:38.6258838Z ==== Alternative Results (includes all methods including those without corresponding source) ====
2024-05-16T09:32:38.6369813Z Alternative Visited Classes 2881 of 2888 (99.76)
2024-05-16T09:32:38.6470437Z Alternative Visited Methods 5687 of 5717 (99.48)
2024-05-16T09:32:38.6531998Z Alternative maximum CRAP score 28

Anonymising the file names is entirely fine; the key items are the numbers of visits, times taken, and number of entities.

[rinkal-chomal7]

Hi Steve,

Below is requested summary data for the altcover:

Summary1: This takes 30mins for processing 50,192,781b of instrumented coverage.

14:37:27 [exec] ... C:\dotnet_coverage\coverage\OAService\coverage.xml.0.acv (50,192,781b)
15:07:19 [exec] 76,348,262 visits recorded in 00:29:40.1763516 (42,888 visits/sec)
15:07:19 [exec] A total of 76,348,262 visits recorded
15:07:19 [exec] Coverage statistics flushing took 0.25 seconds
15:07:19 [exec] Visited Classes 39 of 49 (79.59)
15:07:19 [exec] Visited Methods 181 of 245 (73.88)
15:07:19 [exec] Visited Points 1890 of 2715 (69.61)
15:07:19 [exec] Visited Branches 356 of 747 (47.66)
15:07:19 [exec] Maximum CRAP score 56
15:07:19 [exec] ==== Alternative Results (includes all methods including those without corresponding source) ====
15:07:19 [exec] Alternative Visited Classes 40 of 49 (81.63)
15:07:19 [exec] Alternative Visited Methods 185 of 264 (70.08)
15:07:19 [exec] Alternative maximum CRAP score 56

Summary2: This takes 1min for processing 2,50,36,175b of instrumented coverage.

11:33:56 [exec] ... C:\dotnet_coverage\coverage\OTService\coverage.xml.0.acv (2,50,36,175b)
11:34:39 [exec] 5,05,11,243 visits recorded in 00:00:41.7203742 (12,10,709 visits/sec)
11:34:39 [exec] A total of 5,05,11,243 visits recorded
11:34:39 [exec] Coverage statistics flushing took 1.87 seconds
11:34:41 [exec] Visited Classes 53 of 844 (6.28)
11:34:41 [exec] Visited Methods 284 of 5421 (5.24)
11:34:41 [exec] Visited Points 3559 of 42099 (8.45)
11:34:41 [exec] Visited Branches 833 of 22997 (3.62)
11:34:41 [exec] Maximum CRAP score 8742
11:34:41 [exec] ==== Alternative Results (includes all methods including those without corresponding source) ====
11:34:41 [exec] Alternative Visited Classes 54 of 895 (6.03)
11:34:41 [exec] Alternative Visited Methods 294 of 6127 (4.8)
11:34:41 [exec] Alternative maximum CRAP score 8742

I have tried the coverage instrumentation by excluded any third party assemblies but it excluded everything using
--assemblyFilter and generated empty coverage report because we have .exe and not the dlls for instrumenting the coverage.

[SteveGilham]

So, all the time is being taken reading (decompressing and indexing) the recorded visit data; with code base not much larger than AltCover even in the worst case.

In the slower case, the deflate compression has been extremely efficient, reducing each ~40 byte visit record to just over 5 bits, an indication that a lot of repeat visits are happening; this may account for the amount of processing time needed to unpack it all.

There are two approaches that can be taken to reduce the amount of processing done in this stage.

If you're only interested in whether code has been visited, rather than how often. the --single instrumentation option will skip recording repeat visits.

Alternatively, we can take advantage of the fact that you are not running tests under vstest.exe - which program's habits forced the eager writing of coverage data as default - and use --defer which delays writing any of the coverage data to the ProcessExit event of the instrumented code, and then writes the data in properly indexed form; turning n instances of {module,line,1} to a single {module,line,n}. As in this case values of n are likely to be in the hundreds of thousand, that will be a massive reduction.

---

On the use of filters, which may not be relevant, and certainly don't look to have a significant effect in this case

If there are any unwanted assemblies (modules) in the coverage report, then they can be skipped by name; a usual one for me is that FSharp.Core will be there, and dominating actual code of interest, so --assemblyFilter "FSharp\.Core" - the \ because the string is interpreted as a regex - or even just --assemblyFilter "FSharp" because partial matches are enough to filter.

Alternatively, if your code of interest all has a common name part like MyCompany, then --assemblyFilter "?MyCompany" will exclude all non-matching ones.

[SteveGilham]

For the moment I shall close this as duplicate of Issue #175