Fails to log coverage when using Debugger.IsAttached

[sixfiveohtwo]

Installed product versions

• Visual Studio: 2019 Community 16.8.4
• This extension: 1.1.78
• Xunit 2.4.1, .net core 3.1

Description

There's not really a short explanation for this problem. Let's say we have a suite of unit tests and as well as narrow integration tests. i.e. isolated tests that communicate with a single service. We want these integration tests to be manually executable but we don't want them to run by default and we don't want them to run in the pipeline. nunit provides the [Explicit] attribute for this. MSTests provides no functionality and xunit has a kludge that people in the field are relying on. It goes as follows:

Inherit the [Fact] attribute and apply to check to see if the debugger is attached. If the debugger is attached, then don't skip the test. This allows tests flagged with this new attribute to be skipped by default and in the pipeline but they'll execute when you choose Debug Tests. This is ugly but it's a real solution that people are suggesting are relying upon as it's the only way to achieve this in xunit.

This solution works but FCC produces incorrect results.

Consider the code below:

using Xunit;
using System.Diagnostics;

namespace Test
{
    public class UnitTest1
    {
        [Fact]
        public void TestLucky()
        {
            Assert.Equal(7, Numbers.Lucky());
        }

        [DebugOnly]
        public void TestUnlucky()
        {
            Assert.Equal(13, Numbers.Unlucky());
        }
    }

    public class DebugOnlyAttribute : FactAttribute
    {
        public DebugOnlyAttribute()
        {
            if (!Debugger.IsAttached)
            {
                Skip = "Only running in interactive mode.";
            }
        }
    }

    public class Numbers
    {
        public static int Lucky()
        {
            return 7;
        }

        public static int Unlucky()
        {
            return 13;
        }
    }
}

Current behavior

Both of these tests execute and pass. However, Fine Code Coverage reports that Unlucky() and its test are never executed.

Expected behavior

Code coverage should be reported correctly

Side Notes

I've tried my best to minimize this to the smallest amount of code that can demonstrate this with certainty. Perhaps interestingly, it's not the inheritance that causes the problem. It's the IsAttached check. If you remove that block and then execute in debug, FCC reports coverage as expected.

[sixfiveohtwo]

Well, after porting my project to nunit and using [Explicit], I get the same type of behaviour. Unfortunately, FCC is broken for me when it comes it comes to any form of optional tests. In fact it's all kinds of temperamental for me. This is just the only issue I could define with a repeatable test.

[FortuneN]

Please do post the other issues you have.
Please attach repros; they make things so much simpler

[tonyhallett]

FCC does not have access to the tests that have run/skipped. It executes the coverage tool providing the test dll and there is no concept of a debugger.

Can you instead provide an environment variable that a custom FactAttribute has access to ?
Perhaps this can help https://www.nuget.org/packages/Xunit.SkippableFact
This suggests that it is possible to specify in runsettings.