Errors not reported with duplicated scenarios names

[gaeljw]

Context

I believe the issue might not be related to cucumber-jvm-scala because I only reproduce it when using sbt as build tool rather than maven but I feel it's better to enter an issue here in case someone have the issue in the future.
We'll link to another issue in another project if the fix is not part of this project.

Issue

Given we have two scenarios with the same name (Scenario: a scenario name) in the same file
When I run sbt test
And one of them is failing but not the other
Then the result is "SUCCESS" instead of "ERROR"

It seems that only the result of the first test is considered, thus depending on order of tests the result is different.

Reproduction

Clone https://github.com/gaeljw/cucumber-jvm-scala-issue-duplicate-scenarios/tree/master and run sbt test

What about Maven?

As mentionned, I cannot reproduce with the same project (Scala) using Maven (https://github.com/gaeljw/cucumber-jvm-scala-issue-duplicate-scenarios/tree/maven)

[mpkorstanje]

Sounds like there is a bug in the integration between SBT and JUnit. I'm guessing that SBT assumes that tests are class+method based and thus always unique. You can probebly reproduce this using only JUnit by creating a ParentRunner implementation that has two identically named tests.

[gaeljw]

Yes, you're absolutely right 👍

I tracked down the issue and it comes to the integration between sbt and JUnit.
I will open an issue on sbt side.
We could workaround this somehow by assigning a unique description to the TestNotifier in
JUnitReporter.java#L90 in cucumber-junit but I guess it doesn't make much sense if this works fine in Maven/Java world.

[gaeljw]

Relates to sbt/junit-interface#84.

I will close this one.

[mpkorstanje]

Looks like you may have to do some convincing here:

sbt/junit-interface#83 (comment)

Pretty sure we are within specs.