throwing SkipException sets iTestResult status to Failure instead of Skip

[xlenz]

TestNG Version 6.13.1

Expected behavior

previously I had in beforeInvocation:

throw new SkipException("Skipped due to previous failure of: " + failedTestName);

So I have TestResult.SKIP in afterInvocation.

Discussion started here: #1611

Actual behavior

After upgrading to 6.13.1 it stopped working, because iTestResult.getStatus() is now TestResult.FAILURE instead of TestResult.SKIP in afterInvocation.

to workaround this I'm setting status explicitly before throwing SkipException:

iTestResult.setStatus(TestResult.SKIP);
throw new SkipException("Skipping the test case");

Is the issue reproductible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

// listener
public class TestNGExecutionListener implements IInvokedMethodListener {
    @Override
    public void beforeInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
        throw new SkipException("skip");
    }

    @Override
    public void afterInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
        iTestResult.getStatus(); // actual Failure (2), expected Skip (3)
    }
}

// add listener
testng.addListener((ITestNGListener) new TestNGExecutionListener());

[ghost]

Setting the iTestResult.setStatus(TestResult.SKIP) before throwing the SkipException in the #beforeInvocation method causes me to see failures if the method is a BeforeXXX.

[klubi]

Is there a plan to fix this, or provide a mechanism that would allow skipping tests via configuration method? Above workaround seems to not work (I'm currently on version 7.0.0).
I'd like to skip tests, not disable them with @Test(enabled=false). Outcome is similar, but in terms of reporting (ex system tests) difference is huge.

[klubi]

Hi

I see this was fixed, and merged to master. Any chance this get's released anytime soon?

[krmahadevan]

ping @cbeust - Please chime in.

[klubi]

@krmahadevan Can I get a confirmation this is fixed in 7.3.0 release?
I can still see tests being marked FAILED instead of SKIP when listener throws SkipException.

[krmahadevan]

@klubi - Are you seeing this not working in 7.3.0 ? This version has been released.

[klubi]

@krmahadevan I'm on 7.3.0, and when my listeners throw SkipException tests are still marked as Failed.

When gradle logging is enabled I see something like this:

11:47:59  Test Suite >  Tests > com.aaa.bbbTest > shouldDoSomething FAILED
11:47:59      org.testng.SkipException
11:47:59  
11:47:59  Test Suite >  Tests > com.aaa.bbbTest > shouldDoSomething SKIPPED

and then I see two records for same method in html test report

[krmahadevan]

@klubi - Do you have a sample that you can share so that we can investigate on this ?

[klubi]

@krmahadevan let me prepare one

[klubi]

@krmahadevan
https://github.com/klubi/vsc-testng7-sample

I re-used sample project I created for issue with VS Code.
I created ShouldSkipTest class, and testSuite.xml, and wrapped it with runSkiptest gradle task.

[klubi]

@krmahadevan any update on this?
It seems to still be an issue.

[sbrannen]

I believe this is effectively a duplicate of spring-projects/spring-framework#26387.

See spring-projects/spring-framework#26387 (comment) for further details.

[krmahadevan]

@klubi - I haven't gotten around to looking at this issue yet. The last few months have been extremely hectic at work front. I hope to get things sorted out at work front in a couple of weeks after which I should have some breather time. Transitioning into a full fledged backend developer at work with zero knowledge on Spring is taking a toll on my time, while still juggling the current role's responsibilities.

[juherr]

@krmahadevan No need to learn Spring. You can have a look at the test cases I added at #2455. They are based on the sample provided by @sbrannen

[krmahadevan]

@juherr - Lol.. NO that was not what I meant. I was meaning to say that since at work, I am trying to transition into a backend developer, I am having to invest a lot of time trying to figure out things in Spring etc., and hence I haven't been able to find a lot of time to get to take a look at this issue. btw thanks for adding that sample. I will take it up from there hopefully over this weekend.

[juherr]

@krmahadevan Oops, sorry I didn't catch it. BTW I've just added some new test cases. I think there are some relations with #2423 too.

[sbrannen]

@krmahadevan, thanks for pushing the fix for this!

I don't see a milestone assigned. Is this expected to be included in a 7.3.x release?

[krmahadevan]

@sbrannen - 7.3.0 is already released. This should be part of 7.4.0 (which would be the next upcoming release for TestNG)

@juherr - I think we should start leveraging the milestone capability in github so that our users exactly know as to when would a fix be available. Any thoughts on how to go about doing this ?

[krmahadevan]

@sbrannen - Thank you for bringing up this. I will start adding this milestone for all the issues that would be part of 7.4.0 going forward.

[juherr]

I think we should start leveraging the milestone capability in github so that our users exactly know as to when would a fix be available. Any thoughts on how to go about doing this ?

@krmahadevan Good idea 👍

[krmahadevan]

@juherr - As a first step, i went ahead and added 7.4.0 as a milestone to all defects that we have fixed so far after we released 7.3.0. Going forward when we close an issue, we should ensure we update the milestone to whatever is the current upcoming version. That way this mapping would be there all the time.

[sbrannen]

Glad to you see adopting GitHub milestones.

The community thanks you! 👍