Eclipse JUnit reports inaccurate run count

[dgradl]

Using the eclipse JUnit feature cucumber junit is over-reporting the run count by seemingly double. For example with 4 scenarios of 3 steps (with steps not written yet) each it reports Runs: 24/12. In terms of the double counting it seems to be a matter of the way in which JUnitReporter is sending notifier events to the EachStepNotifier. It doesn't seem to be consistent with JUnit itself, where if a test is ignored, it does a fireTestIgnored without a fireTestStarted.

I also question whether each individual cucumber step (GIVEN, WHEN, THEN) should be counted as a run as opposed to just the scenario or example itself. I generally expect the run count to give me a number of tests invoked (in JUnit it is a reflection of number of @tests). However, its really only a count of steps. After writing several scenarios with multiple GIVEN-AND and THEN-ANDs on a project I am working on it reported 390 runs, but its really only 90 some tests executed.

[aslakhellesoy]

What cucumber-jvm version are you running?

Cucumber features are reported in a JUnit hierarchy:

Feature -> Scenario -> Step
Feature -> Scenario Outline -> Example Row -> Step

JUnit will report each leaf as a test, and I'm not sure there is a way around that.

Regarding fireTestIgnored without a fireTestStarted - that sounds correct.

The JUnit runner still needs a lot of improvement, so any contributions are welcome.

[dgradl]

I have 1.0.0.RC20 from maven for core and java, combined with a git clone of the JUnit project (which is RC21 I believe). I agree there may not be a way around the leafs being reported as tests. But I am looking into these things. I wanted to report a bug to track and it to get feedback from the community as to whether (if it were possible) it should not count steps as tests.

[aslakhellesoy]

I don't think it's a problem to count leafs (steps) as tests. However, counting should obviously be correct and show 12/12 instead of 24/12.

I'm an IDEA user, so I haven't tested this on Eclipse yet. What exactly is a "run"? Is that something Eclipse reports? How is it different from a "test"?

[dgradl]

With JUnit tests a Run equates to a test execution. So for example if you have a parameterized test (@RunWith(Parameterized.class)) with JUnit4 and there is a single method annotated with @test, but a set of 8 parameters. And it will list runs as 8. So it is executions of a test. I don't really think the steps are actually test executions in themselves, it takes the whole Given-When-Then to form an actual test, so that is why I don't like it counting each as a run. However, it is valuable to treat them as leafs in the execution tree, and to visually see where along that path it failed (e.g. an exception being thrown in a When step). So it may be something that is unavoidable. The main reason it even affected me is that I was trying to provide someone with a count of how many tests were being generated by the features and I could not use the Runs count to report that value. However, I can also count the number of scenario / example runs and derive that.

[mmalmeida]

I remember having twice the count in the tests in Eclipse a while ago. Now I don't remember what I changed to get the correct count - it was definitely a random thing, not a feature.
Perhaps I posted the explanation either here or in the mailing list - I'll try to find it.

[aslakhellesoy]

I think we all agree that it's important to have a leaf per step in order to see how execution went.

What exactly are you suggesting should be done in this ticket?

[dgradl]

Leaf per step is fine. Submitted a pull request to address the double counting.

[aslakhellesoy]

I'm reopening this because it still seems to be double quoting: https://groups.google.com/d/msg/cukes/huCNeimgU4w/UJ4-qhjXRFgJ

[yurypetrov]

I've just started using cucumber-jvm. Works fine for us so far, thank you!

However, I am not sure why we need to have a leaf per step (not per scenario)? If we have a leaf per scenario will it be much worse in terms of understanding what step the test failed at? Any other reasons?

Maybe there's some thread that describes this in more details?

Thanks!

[aslakhellesoy]

I agree completely that it was a bad decision to make steps be junit leaves. In a future version the leaves will be scenarios. If a scenario fails there will be enough info in the stack trace to figure out what step/stepdef failed. To improve the context we could also add a textual description of the scenario with steps in the exception message. It could even have ANSI colours in them, so that exceptions printed in ANSI terminals would show colours for exceptions. We would have to figure out a way to only send colours to TTY, not files.

[yurypetrov]

Thanks for the reply! Glad to hear we will move to scenario leaves in future!

Aslak, I suppose it will take time to make this change?
Are there any temporarily solutions you could recommend?

[aslakhellesoy]

No, no temporary solutions I'm afraid. And yes, it will take a few months at least. It might happen before or after a new gherkin parser we're working on.