Ignore random coverage changes

[wilzbach]

We seem to have gotten a non-deterministic behavior into the test suite, which is why the coverage currently changes for non-related changes.
There are quite many recent PR with this behavior (see e.g. #6871 or #6878).

This adds a workaround against non-deterministic coverage changes analogous to Phobos.

CC @CyberShadow

[CyberShadow]

Unlike with the Phobos case, I don't see any effort to understand and fix why the coverage is shifting.

[WalterBright]

I have no idea why #6878 should have random results. It's completely deterministic.

[CyberShadow]

I have no idea why #6878 should have random results. It's completely deterministic.

It's caused by something else in the test suite. For now we don't know what.

Unlike dlang/phobos#5375 though, where the non-determinism was occuring in error handling of parallelism code (so it was at least somewhat understandable), here the non-determinism occurs in compiler logic, which should be completely deterministic. If I were to guess, there is a race condition in the test suite or the way coverage reports are generated.

@wilzbach The coverage analysis is done with -j1, right?

[wilzbach]

@wilzbach The coverage analysis is done with -j1, right?

Nope, -j2 respective line.

If I were to guess, there is a race condition in the test suite or the way coverage reports are generated.

Yep I thought so as well, but I couldn't think of an easy way to find the real root cause. Hence this "hack", but I agree that it's probably worthwhile to invest a bit of time into finding the real problem here.

[CyberShadow]

Nope, -j2 respective line.

Um... so what happens when two dmd instances run at the same time, and try to save their coverage reports?

[wilzbach]

Um... so what happens when two dmd instances run at the same time, and try to save their coverage reports?

I guess "undefined"?
Let's see how the impact of using -j1 is:

#6882

[wilzbach]

Let's see if this helps with the random failures. If not, I guess we can revert it to speed it up later.

It didn't help, see e.g. #6898

There's also a test which covers this: make test_results/compilable/test13226.d.out
However only without argument permutation. Maybe something is wrong there?

[wilzbach]

Closing this for now.