From 6 seconds to almost forever

[ericsink]

I'm logging a separate issue for this because I'm not sure it is the same as #56 or #51, although it might be.

In this repo:

https://github.com/ericsink/sawdust

If I cd into the sd_tests directory and type dotnet test, it takes about six seconds (on my machine).

OTOH, when I did:

dotnet add package AltCover
dotnet test /p:AltCover=true

It ran for hours before I killed it, consuming a large amount of disk space.

Anyway, I thought perhaps having another repro case might be helpful.

[SteveGilham]

Thanks for this.

[SteveGilham]

It does look like the symptoms of #51.

That using /p:AltCoverSingle=true drops the disk space consumption to practically nothing (stable at 4kb) suggests there's a small core of code being executed in a tight spin, sufficient to defeat the compression of the output.

My gut feel at the slowdown is that there is sufficient multi-threading going on in these tests that the serializing of the coverage hit recording is dominating the run-time (probably that tight loop interfering with everything else). Addressing that without unfixing #42 is going to be interesting.

[ericsink]

That makes sense. I do believe there is a fair amount of multi-threading in play.

[SteveGilham]

Using dotnet test /p:AltCover=true -v m /p:AltCoverAssemblyExcludeFilter=NUnit /p:AltCoverForce=true /p:AltCoverSingle=true the end result is

Test execution time: 1.4887 Hours
  ... C:\Users\steve\Documents\GitHub\sawdust-master\sd_tests\coverage.xml.0.acv (182,998b)
  15,337 visits recorded in 00:00:00.2920417 (52,516 visits/sec)
  A total of 15,337 visits recorded
  Coverage statistics flushing took 1.69 seconds
  Visited Classes 73 of 79 (92.41)
  Visited Methods 925 of 980 (94.39)
  Visited Points 13019 of 14703 (88.55)
  Visited Branches 2695 of 3607 (74.72)

  ==== Alternative Results (includes all methods including those without corresponding source) ====
  Alternative Visited Classes 73 of 84 (86.9)
  Alternative Visited Methods 925 of 1014 (91.22)

I think this is going to end up with a number of options available to tune the recording to the individual scenario, rather than trying for a one-size fits all.

[ericsink]

Well I will certainly let you prioritize your own work, but getting AltCover working with the sawdust code base isn't terribly important for me. I submitted it here just in case it was useful to you as a reproduceable test case. But one of the things I say in the README for the sawdust repo is that it isn't a code base I currently plan to work on very much. So if you are considering fixes or enhancements that are likely to improve AltCover for sawdust without much chance of benefiting other projects, I would suggest maybe not. :-)

Thanks!

[SteveGilham]

On the assumption that this is the same underlying issue as #51, I now have something tractable to work on for that issue. And most of the remaining roadmap items are just boring XML manipulation.

[SteveGilham]

Having made a number of experimental forays, the root cause is that the set of tests presented are compute bound, and make in excess of 1.3E10 recordable visits, and hence the recording overheads dominate, even in experiments when buffered to memory rather than promptly written to file. Experimental results suggest that the best-case scenario would be "only" about 100x slowdown, and be limited by whether the in-memory record can be serialized out in the short time window available to the .net core ProcessExit handler; sawdust plus tests (excluding the NUnit code) is around the size where even the lightest weight formats are not reliably completely written in the available time.

For the record, the 1.3E10 number is a couple of orders of magnitude more coverage visits than I've seen even in large-scale system testing during my career, admittedly in systems that were constrained by I/O in its broadest sense.

[ericsink]

Interesting. I guess the sawdust test suite is kind of a pathological case.

I was hoping sawdust as an example might have been more helpful to you, but alas...

[SteveGilham]

Release v5.3.675 makes some significant speed-ups for dotnet test cases which have heretofore been limited by the data collection mechanism used to work around vstest.console's hard guillotine at process exit.

On my machine, the test set takes about 35 minutes to record 12,999,758,207 visits, compared with 13s without instrumentation, for a factor of only ~160 slowdown.