JUnit formater doesn't report Scenario outline example with -x argument

[floreal]

Hello,
I noticed a little annoying bug. consider my Rakefile:

require 'rubygems'
require 'rake'
require 'cucumber/rake/task'

# some other tasks

namespace :cucumber do
  Cucumber::Rake::Task.new(:report) do |t|
    t.cucumber_opts = "-x -p #{browser} -f junit --out report -f pretty --no-color"
  end

  Cucumber::Rake::Task.new(:pretty) do |t|
    t.cucumber_opts = "-x -p #{browser}"
  end
end

The cucumber proffiles only define environment variable to set selenium url and desired browser capabilities

When i execure rake cucumber:report, only Scenarios are reported to the JUnit result report, Scenario Outlines examples aren't.
When I remove the -x flag they are reported.

[floreal]

I managed to see what happen actually: when -x argument is passed, whenever an error (even error on asseertions) occurs in a scennario outline case, the other cases of the outline wont execute (and will be considerated as failed)

[tobhahn]

Here is a scenario to reproduce the error on today's master.

Steps to reproduce:

• Save the following feature into legacy_features/junit_formatter_expand.feature:

Feature: JUnit output formatter
  In order for developers to create test reports with ant
  Cucumber should be able to output JUnit xml files

  Background:
    Given I am in junit
    And the tmp directory is empty


  Scenario: Expand scenario outline into one passing and one failing scenario
    When I run cucumber --format junit --expand --out tmp/ features/scenario_outline.feature
    Then it should fail with
      """

      """
    And "fixtures/junit/tmp/TEST-features-scenario_outline.xml" with junit duration "0.005" should contain
      """
      <?xml version="1.0" encoding="UTF-8"?>
      <testsuite errors="0" failures="1" name="Scenario outlines" skipped="0" tests="2" time="0.005">
      <testcase classname="Scenario outlines.Using scenario outlines" name="Using scenario outlines (outline example : | passing |)" time="0.005">
      </testcase>
      <testcase classname="Scenario outlines.Using scenario outlines" name="Using scenario outlines (outline example : | failing |)" time="0.005">
        <failure message="failed Using scenario outlines (outline example : | failing |)" type="failed">
          <![CDATA[Scenario Outline: Using scenario outlines

      Scenario: |failing|

      Given a failing scenario

      Message:
      ]]>
          <![CDATA[ (RuntimeError)
      features/scenario_outline.feature:4:in `Given a <type> scenario']]>
        </failure>
      </testcase>
      </testsuite>

      """

• Run bin/cucumber legacy_features/junit_formatter_expand.feature.

Expected results:
No error.

Actual results:
The scenario fails with

Using the default profile...
......F

(::) failed steps (::)

expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuite errors=\"0\" failures=\"1\" name=\"Scenario outlines\" skipped=\"0\" tests=\"2\" time=\"0.005\">\n<testcase classname=\"Scenario outlines.Using scenario outlines\" name=\"Using scenario outlines (outline example : | passing |)\" time=\"0.005\">\n</testcase>\n<testcase classname=\"Scenario outlines.Using scenario outlines\" name=\"Using scenario outlines (outline example : | failing |)\" time=\"0.005\">\n  <failure message=\"failed Using scenario outlines (outline example : | failing |)\" type=\"failed\">\n    <![CDATA[Scenario Outline: Using scenario outlines\n\nScenario: |failing|\n\nGiven a failing scenario\n\nMessage:\n]]>\n    <![CDATA[ (RuntimeError)\nfeatures/scenario_outline.feature:4:in `Given a <type> scenario']]>\n  </failure>\n</testcase>\n</testsuite>\n"
     got: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuite errors=\"0\" failures=\"0\" name=\"Scenario outlines\" skipped=\"0\" tests=\"0\" time=\"0.005\">\n</testsuite>\n" (using ==)
Diff:
@@ -1,20 +1,4 @@
 <?xml version="1.0" encoding="UTF-8"?>
-<testsuite errors="0" failures="1" name="Scenario outlines" skipped="0" tests="2" time="0.005">
-<testcase classname="Scenario outlines.Using scenario outlines" name="Using scenario outlines (outline example : | passing |)" time="0.005">
-</testcase>
-<testcase classname="Scenario outlines.Using scenario outlines" name="Using scenario outlines (outline example : | failing |)" time="0.005">
-  <failure message="failed Using scenario outlines (outline example : | failing |)" type="failed">
-    <![CDATA[Scenario Outline: Using scenario outlines
-
-Scenario: |failing|
-
-Given a failing scenario
-
-Message:
-]]>
-    <![CDATA[ (RuntimeError)
-features/scenario_outline.feature:4:in `Given a <type> scenario']]>
-  </failure>
-</testcase>
+<testsuite errors="0" failures="0" name="Scenario outlines" skipped="0" tests="0" time="0.005">
 </testsuite>
 (RSpec::Expectations::ExpectationNotMetError)
./legacy_features/step_definitions/cucumber_steps.rb:110:in `/^"([^"]*)" with junit duration "([^"]*)" should contain$/'
legacy_features/junit_formatter_expand.feature:16:in `And "fixtures/junit/tmp/TEST-features-scenario_outline.xml" with junit duration "0.005" should contain'

Failing Scenarios:
cucumber legacy_features/junit_formatter_expand.feature:10 # Scenario: one feature, expand one scenario outline, two examples: one passing, one failing

1 scenario (1 failed)
5 steps (1 failed, 4 passed)

[shyam-prabhakar]

Any plans on fixing this, this seems like a fairly big issue for folks who want to use the 'expand' option with the junit formatter for scenario outlines.

[cjapes]

This is big issue for the people that are using cucumber with Jenkins, that depend on that junit output.
Any plans on fixing this?

[rtyler]

This is affecting me as well, since I was probably one of the last people to touch the JUnit formatter, I'll look into fixing this up.

[mattwynne]

@rtyler do you think you're still likely to fix this?

[mattwynne]

Hands up if you still care about this issue.

[mattwynne]

@brasmusson @richarda either of you fancy a crack at this one?

[richarda]

Sure!
On Fri, Feb 13, 2015 at 5:48 PM Matt Wynne notifications@github.com wrote:

@brasmusson https://github.com/brasmusson @richarda
https://github.com/richarda either of you fancy a crack at this one?

—
Reply to this email directly or view it on GitHub
cucumber/common#124 (comment).

[tooky]

Closed in favour of #808

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.