beforeConfiguration() listener method should be invoked for skipped configurations as well

[bj-9527]

TestNG Version

7.5

Expected behavior

The TestNG listener method beforeConfiguration() should be invoked for skipped configurations as well.

Actual behavior

beforeConfiguration() is currently NOT being invoked for skipped configuration methods.

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

public class SampleListener implements ITestListener, IInvokedMethodListener, IConfigurationListener {
  @Override
  public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
    System.out.println("before invocation executed");
  }

  @Override
  public void beforeConfiguration(ITestResult testResult) {
    System.out.println("before configuration executed");
  }
}

@Listeners({SampleListener.class})
public class IssueTest {
  @BeforeClass
  public void beforeClass() {
    int i = 5/0;
  }

  @BeforeMethod
  public void beforeMethod() {

  }

  @Test
  public void issueTest() {
    System.out.println("hello world");
  }
}

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[juherr]

Could confirm that the configuration method is not called?

What is the state of ITestResult when the listener method is called?

[krmahadevan]

@bj-9527 - I am not quite sure I understand this issue.

Here's what I see.

• Your @BeforeClass method threw an exception
• This means that nothing in that class is now going to be eligible for being executed because a higher level configuration failed.
• That explains why the @BeforeMethod was skipped because when it comes to "weightage" a @BeforeClass is more important than a @BeforeMethod
• The test got skipped because there's a configuration failure.
• The listener is being invoked ONLY for the beforeClass() configuration. So IMO, all is well.

The ONLY discrepancy I see is that the beforeInvocation() is being executed even for beforeMethod() which is a SKIPPED configuration. I am not sure as to how this can be interpreted. If a config is skipped should listener invocation for that be skipped too ? Its a bit grey area for me, because one can argue that listeners should be invoked irrespective of the state of a method and the counter argument can be that it should be selective. TBH I don't know the expected outcome.

Please feel free to chime in if I have misunderstood this issue. If this answers your question, please feel to close off this issue.

[krmahadevan]

Could confirm that the configuration method is not called?

@juherr - beforeMethod() is NOT invoked.

What is the state of ITestResult when the listener method is called?

The state of ITestResult is

• STARTED in beforeInvocation() and
• in afterInvocation()
  • beforeMethod() --> SKIPPED
  • beforeClass() --> FAILURE

[juherr]

@krmahadevan Looks good to me.
Did the behavior change in the latest release?

[bj-9527]

@krmahadevan thanks for clarify.
In this case, what I expect is that if the listener will execute, then all listeners method should be execute.
In my case, we will do some initial in before config, then go before invocation, but if the previous before class failed, the only the before invocation will be called, in this scenario, we will lose the initial part, it would cause confused result.

[bj-9527]

@krmahadevan Looks good to me. Did the behavior change in the latest release?

In 6.14.3, the result is the listener method for before method will not execute, in 7.4 or 7.5, it will only call the before invocation

[krmahadevan]

@bj-9527

I made some pretty printing modifications to the listener and here's the output comparison across 7.5, 7.4.0 and 6.14.3

7.5

Version testng-7.5.jar
beforeConfiguration executed for configurationmethod beforeClass() and its status was STARTED
beforeInvocation executed for configurationmethod beforeClass() and its status was STARTED
afterInvocation executed for configurationmethod beforeClass() and its status was FAILURE
beforeInvocation executed for configurationmethod beforeMethod() and its status was STARTED
afterInvocation executed for configurationmethod beforeMethod() and its status was SKIP
beforeInvocation executed for testmethod issueTest() and its status was SKIP
afterInvocation executed for testmethod issueTest() and its status was SKIP

Version testng-7.4.0.jar
beforeConfiguration executed for configurationmethod beforeClass() and its status was STARTED
beforeInvocation executed for configurationmethod beforeClass() and its status was STARTED
afterInvocation executed for configurationmethod beforeClass() and its status was FAILURE
beforeInvocation executed for configurationmethod beforeMethod() and its status was STARTED
afterInvocation executed for configurationmethod beforeMethod() and its status was SKIP
beforeInvocation executed for testmethod issueTest() and its status was SKIP
afterInvocation executed for testmethod issueTest() and its status was SKIP

Version testng-6.14.3.jar
beforeInvocation executed for configurationmethod beforeClass() and its status was SUCCESS
afterInvocation executed for configurationmethod beforeClass() and its status was FAILURE
beforeInvocation executed for testmethod issueTest() and its status was CREATED
afterInvocation executed for testmethod issueTest() and its status was CREATED

Analysis:

• beforeConfiguration() didn't exist for 6.14.3
• beforeInvocation() was NOT invoked for skipped configurations in 6.14.3 and i think this was changed in a later version (I dont know which one)
• beforeConfiguration() method is not being invoked for skipped configurations (This is consistent across 7.4.0 and 7.5)

So now I would need to know what is this bug tracking ?

Is this issue suggesting that beforeConfiguration() is NOT being invoked for skipped configurations ?

If yes, please have this defect updated with the proper description and expectation.

@juherr - I hope this adds clarity to your questions that you were asking for.

[bj-9527]

Is this issue suggesting that beforeConfiguration() is NOT being invoked for skipped configurations ?

Yes, as I described before, when execution comes to before method, all the listener method should be executed.
the expected order is like:

beforeConfiguration executed for configurationmethod beforeClass() and its status was STARTED
beforeInvocation executed for configurationmethod beforeClass() and its status was STARTED
afterInvocation executed for configurationmethod beforeClass() and its status was FAILURE

//this method is expect to call 
beforeConfiguration executed for configurationmethod beforeMethod() and its status was STARTED

beforeInvocation executed for configurationmethod beforeMethod() and its status was STARTED
afterInvocation executed for configurationmethod beforeMethod() and its status was SKIP
beforeInvocation executed for testmethod issueTest() and its status was SKIP
afterInvocation executed for testmethod issueTest() and its status was SKIP

If the beforeClass is not failed, it would like:

What I expect is that normal and abnormal behaviour would be aligned.

[krmahadevan]

@bj-9527 - Ok. I have updated the issue title and description to match the expectation. Please take a look

[bj-9527]

@bj-9527 - Ok. I have updated the issue title and description to match the expectation. Please take a look

Thanks a lot, it becomes more clear, I will try to raise PR later