TestNG reports entire Cucumber test suite as a single <testcase>

[rankinc]

When running cucumber-jvm 1.2.0 with JUnit, Jenkins reports each Cucumber step as an individual testcase. However, switching to use TestNG with RunCukes implemented as:

@CucumberOptions(plugin = "json:target/cucumber-report.json")
public class RunCukesTest extends AbstractTestNGCucumberTests {
}

Jenkins then reports the entire RunCukesTest as a single testcase called "run_cukes".

Cucumber's own report of (e.g.)

5 Scenarios (4 failed, 1 passed)
72 Steps (4 failed, 44 skipped, 24 passed)

is the same with both JUnit and TestNG, so I think the problem must be with the TestNG reporting.

[aslakhellesoy]

Yes, that's right. I don't use TestNG myself, but I'd be happy to accept a pull request fixing this.

[rankinc]

Can you offer any advice on what kind of change would be required, please? I can't claim to understand Cucumber, JUnit or TestNG's internal workings - I typically just invoke them.

[aslakhellesoy]

You'd have to study the source code and tests under cucumber-testng. That's the best advice I can offer.

[rankinc]

That doesn't fill me with confidence. Granted that you don't use TestNG, but I would at least have thought you'd be able to provide some insights into the Cucumber side of the equation.

[aslakhellesoy]

The codebase is small - look here: https://github.com/cucumber/cucumber-jvm/tree/master/testng/src/main/java/cucumber/api/testng

I could have given you more guidance if I had time, but unfortunately I don't.

Pretty much every single patch and pull request to Cucumber-JVM comes from people who have been able to find their way around the codebase on their own.

I realise not everyone who wants something fixed will be able to do this, but the ratio of people who do manage to fix things on their own is high enough to keep the project going forward.

Hope this helps

[rankinc]

Thanks, cucumber-testng is obviously where the new code will need to go. (I found that myself last night). However, I suspect that cucumber-junit might be more useful in determining what this new code might actually be, seeing as JUnit already receives "the knowledge" that I seek...

[aslakhellesoy]

Yeah, I never really understood why people want TestNG integration - except for the ability to run TestNG unit tests in the same invocation as Cucumber.

[rankinc]

As the saying goes - choose a testing framework and then stick with it. And some code-bases have already chosen TestNG.

[brasmusson]

PR #653 proposes changes so that cucumber-testng wraps each Cucumber Feature in one TestNG test (instead of wrapping all the Features in one TestNG test). Sort of the latest version of that PR is actually here (where I fixed my own comments on #653).

I believe that both JUnit and TestNG provides information through two paths, result files and notifications. When used from IDE, the IDE usually integrates with the notifications from the test framework. Jenkins more often reads result files. In case of JUnit, Jenkins can parse the JUnit style result files that the JUnit-formatter of Cucumber-JVM produces. I do not know the standard way of using Jenkins and TestNG. However #621 proposes to add a TestNG-formatter to Cucumber-JVM. That PR is a bit outdated, since some of the constructs taken from the JUnit-formatter (numbering of Scenario Outline scenarios, for instance), since than has been changed in the JUnit-formatter.

[rankinc]

Sort of the latest version of that PR is actually here (where I fixed my own comments on #653).

Thanks, this is interesting. Although it does beg the question - "Why wasn't this work ever pulled?"

[aslakhellesoy]

@rankinc I guess it slipped under the radar so far.

[brasmusson]

I brought the two pending TestNG PR:s (#653 and #621) up to date with the current tip of the master; #817 and #818. I would be great if you evaluated them as a TestNG user @rankinc, neither I am using TestNG myself.

[RomanIovlev]

Is my issue related to this one? What actual status of it? (as I can see PRs 653 and 621 closed but issue still open)
My problem:
If I run tests using TestNG runner I have no steps in my IntelliJIdea view (and can't get them in report)
But if I use JUnit runner I can see all steps
JUnit runner
@RunWith(Cucumber.class)
@CucumberOptions()
public class JUnitRunner { }
Results: http://pix.my/zSSImRsE

TestNG runner
@CucumberOptions()
public class PharmCucumberTestSuite extends AbstractTestNGCucumberTests { }
Results: http://pix.my/kr8B9XNO

[brasmusson]

In Cucumber-JVM v1.2.3 and later, each feature should be executed as a separate TestNG test when extending AbstractTestNGCucumberTests.

[mikejramsey56]

I am running cucumber-java, cucumber-testng v1.2.4 with testng v6.9.10. I am using the TestNG runner. Tagging in the feature files works fine (only the selected features/scenarios execute). But the reports are abysmal. I personally don't have a problem with the JUnit test runner but my client is set on cucumber and TestNG. Cucumber is doing its part by running tagged tests. I'll try to dig around in the TestNG code to see what can be done.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.