NPE while running empty scenario via AbstractTestNGCucumberTests

[fslev]

Steps to reproduce:

1. Create empty scenario

Feature: Feature with empty scenario

  Scenario: Empty scenario

2.) Create a TestNG class for running feature from above:

@RunWith(Cucumber.class)
@CucumberOptions(features = "src/test/resources/features/EmptyScenario.feature",
        glue = {"com.cucumber.utils"}, plugin = {"pretty", "junit:target/junit/cucumber-reports.xml",
        "json:target/cucumber-report/report.json", "timeline:target/cucumber-report/timeline"},
        tags = {"not @Ignore", "not @ignore"})
public class CucumberUtilsTest extends AbstractTestNGCucumberTests implements ITest {

    private ThreadLocal<String> testName = new ThreadLocal<>();

    @Override
    @DataProvider(parallel = true)
    public Object[][] scenarios() {
        return super.scenarios();
    }

    @BeforeMethod
    public void BeforeMethod(Method method, Object[] testData, ITestContext ctx) {
        if (testData.length > 0) {
            testName.set(testData[0].toString());
            ctx.setAttribute("testName", testName.get());
        } else
            ctx.setAttribute("testName", method.getName());
    }

    @Override
    public String getTestName() {
        return testName.get();
    }
}

3. Run test

Actual:

java.lang.NullPointerException
	at io.cucumber.testng.TestCaseResultListener.createUndefinedException(TestCaseResultListener.java:116)
	at io.cucumber.testng.TestCaseResultListener.getError(TestCaseResultListener.java:86)
	at io.cucumber.testng.TestNGCucumberRunner.runScenario(TestNGCucumberRunner.java:126)
	at io.cucumber.testng.AbstractTestNGCucumberTests.runScenario(AbstractTestNGCucumberTests.java:29)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:134)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:597)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:173)
	at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:77)
	at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:15)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

[fslev]

Running via JUnit, it seems no error is thrown:

Scenario: Empty scenario # src/test/resources/features/EmptyScenario.feature:3
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.584 s - in Feature with empty scenario

[mpkorstanje]

The empty scenario somewhat unexpectedly gets the state "Undefined". This unexpected because by definition an empty test should pass. As a result however the TestCaseResultListener probably expects there to be a skipped step.

Now I'm guessing the TestCaseState defaults to Undefined when there are no step results because the result is literally undefined, but that is not the meaning of undefined state.

So ideally we fix that and default to passed. But that will need a major release so the next best thing would be to fix the listeners.