LivingDoc Shows Test Failure for Pending Scenario

[sikerlatte]

SpecFlow Version

3.9.74

Which test runner are you using?

NUnit

Test Runner Version Number

4.1.0

.NET Implementation

equal or greater .NET Framework 4.6.1

Project Format of the SpecFlow project

Classic project format using <PackageReference> tags

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Command line – PLEASE SPECIFY THE FULL COMMAND LINE

SpecFlow Section in app.config or content of specflow.json

{
"stepAssemblies": [
{ "assembly": "Core.TestManagement" }
],
"NRetrySettings": {
"maxRetries": 1,
"applyGlobally": true
},
"runtime": {
"missingOrPendingStepsOutcome": "Ignore"
}
}

Issue Description

LivingDoc is reporting pending tests as a failure rather than Ignored, even though we have specified that outcome in specflow.json.

We run our tests via GitHub actions and execute several feature files at once. In this case, we are running five tests: two of them should be pending, and the other three correctly show as passing on the LivingDoc report. But the pending ones show as "fail."

We set tests as pending by adding an @exclude tag above the Scenario block. In a [BeforeScenario] hook, we correctly identify those with @exclude and call scenarioContext.Pending().

This gives us an exception in our GitHub action report, shown below. It occurs right after all the steps get skipped due to previous errors.

TechTalk.SpecFlow.PendingStepException : One or more step definitions are not implemented yet.
Stack Trace:
at TechTalk.SpecFlow.ScenarioContext.Pending()

So it seems like the PendingStepException is correctly thrown, but LivingDoc is not picking this up.

Steps to Reproduce

1. Add custom @exclude tag above Scenario.
2. Run GitHub action pipeline.
3. Download LivingDoc and examine test results.

Link to Repro Project

No response

[dbaggott]

+1 on this being an issue

[dbaggott]

@sikerlatte , is there a workaround that you are aware of?

[smcthompson]

+1 for me as well. Also of note, I am using xUnit.

[DeLaphante]

I would advised all to not use Specflow+ Runner or LivingDoc as its currently not being supported or developed - migrate to MsTest or Nunit and also update to latest .net 8+ and use third party reporting libraries such as extent report.

Specflow is working perfectly fine for me with the above changes in the following daily github actions repo -> https://github.com/DeLaphante/CynkyAutomation/actions