Listeners may invoked not by the definition order / Have a way to order listeners

[ReinhardXu2007]

TestNG Version

6.9.13.6

Expected behavior

Listeners are invoked by the definition order.

Diagnosis

In TestRunner.initListeners, classes of listeners were put into a HashSet which would lost the order.

    Set<Class<? extends ITestNGListener>> listenerClasses = Sets.newHashSet();
    Class<? extends ITestNGListenerFactory> listenerFactoryClass = null;

    for (IClass cls : getTestClasses()) {
      Class<?> realClass = cls.getRealClass();
      ListenerHolder listenerHolder = findAllListeners(realClass);
      if (listenerFactoryClass == null) {
        listenerFactoryClass = listenerHolder.listenerFactoryClass;
      }
      listenerClasses.addAll(listenerHolder.listenerClasses);
    }

Variable "listenerClasses" could be a LinkedHashSet.

Is the issue reproductible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

[juherr]

Good point, thanks for the report.

Currently, TestNG doesn't specify anything about the order of listeners and they are supposed to be independent.

But 👍 because I already had to manage dependency between listeners and it was a pain.
Sadly, the fix won't be so easy.
Maybe we can replace the HashSet by a LinkedHashSet as a first step but there are many ways to declare a listener and the order will be still strange.

[ReinhardXu2007]

It would help a lot.
It seems that listeners from ServiceLoader and XmlSuite are loaded by the definition order.

[cbeust]

The way I've been solving similar problems is by allowing listeners to specify a priority number, or a weight and then sort them by weight before invoking them. You can be backward compatible by providing a default weight for listeners that don't specify any.

Not sure it's worth fixing, though, I'll let @juherr decide if he wants to tackle this.

[juherr]

sort them by weight before invoking them

But how do you sort listeners?

I'll let @juherr decide if he wants to tackle this

I'd like to keep the issue open. But I'm not able to work on it for the moment

[krmahadevan]

@juherr - Considering the fact that there are ways of ensuring that the listeners are invoked in a user defined order from outside of TestNG, do you think this still needs to be fixed from within TestNG ?

Here's a blog post that I created which shows how to do this from outside of TestNG

https://rationaleemotions.wordpress.com/2016/12/26/dears-listeners-stand-in-q-please/

[sbabcoc]

To enforce a specified invocation order, I created a ListenerChain listener that provides similar behavior to what you can achieve with the JUnit RuleChain. Test classes that need this sort of ordered listener behavior implement a ListenerChainable interface, which declares a method that ListenerChain calls to allow the test class to add its listeners to the chain.

ListenerChain supports listeners that implement any of the following interfaces: ISuiteListener, ITestListener, IClassListener, IInvokedMethodListener, IConfigurationListener2. Methods declared by other listener interfaces will not be invoked.

NOTE: The order in which listener methods are invoked is determined by the order in which listener objects are added to the chain. Listener before methods are invoked in last-added-first-called order. Listener after methods are invoked in first-added-first-called order.
NOTE: Only one instance of any given listener class will be included in the chain.

[krmahadevan]

@sbabcoc - Perhaps posting a sample of what you are talking about would help people get a better idea of the solution ?

[sbabcoc]

https://github.com/sbabcoc/listener-chain

[sbabcoc]

These files are from a larger TestNG foundation library, but I'm not ready to open that up just yet.

[sbabcoc]

The library I mentioned previously has been published: https://github.com/Nordstrom/TestNG-Foundation

[juherr]

@sbabcoc Could you update https://github.com/cbeust/testng/wiki/3rd-party-extensions ? :)

[sbabcoc]

@juherr Done

[krmahadevan]

Fixed via #2737