Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a message based JUnit XML Formatter #2638

Merged
merged 17 commits into from
Dec 27, 2022

Conversation

mpkorstanje
Copy link
Contributor

@mpkorstanje mpkorstanje commented Nov 10, 2022

🤔 What's changed?

Replace the JUnitFormatter with the message based implementation from
cucumber-junit-xml-formatter.
This achieves several goals:

  • Move the internal code base away from the events from the plugin module.
  • Extract common parts of Cucumber into modules that can be shared.

This results in several changes/improvements:

  1. Change the long naming strategy to include the example group index
Feature: Examples Tables
  Scenario Outline: Eating cucumbers
    Given there are <start> cucumbers
    When I eat <eat> cucumbers
    Then I should have <left> cucumbers
    Examples: These are passing
      | start | eat | left |
      |    12 |   5 |    7 |
      |    20 |   5 |   15 |
    Examples: These are failing
      | start | eat | left |
      |    12 |  20 |    0 |
      |     0 |   1 |    0 |

This feature previously contained the examples

 - Examples Tables - Eating cucumbers - These are passing - Example #1
 - Examples Tables - Eating cucumbers - These are passing - Example #2
 - Examples Tables - Eating cucumbers - These are failing - Example #1
 - Examples Tables - Eating cucumbers - These are failing - Example #2

And will now contain the examples:

 - Examples Tables - Eating cucumbers - These are passing - Example #1.1
 - Examples Tables - Eating cucumbers - These are passing - Example #1.2
 - Examples Tables - Eating cucumbers - These are failing - Example #2.1
 - Examples Tables - Eating cucumbers - These are failing - Example #2.2
  1. Rename the suite from io.cucumber.core.plugin.JUnitFormatter to Cucumber
<testsuite failures="0" name="Cucumber" skipped="0" errors="0" tests="1" time="0">
  1. When a scenario is skipped or failed the executed steps are always included in the
    system-out element rather than the skipped or failed element.
<testcase classname="Examples Tables" name="eating cucumbers - These are failing - Example #2.1" time="0.007">
<failure type="AssertionError" message="Expected values to be strictly equal:

-8 !== 0
">
<![CDATA[
<a stacktrace here>
]]>
</failure>
<system-out><![CDATA[
Given there are 12 cucumbers................................................passed
When I eat 20 cucumbers.....................................................passed
Then I should have 0 cucumbers..............................................failed
]]></system-out>
</testcase>
<testcase classname="Skipping scenarios" name="Skipping from a step causes the rest of the scenario to be skipped" time="0.005">
<skipped/>
<system-out><![CDATA[
Given a step that skips.....................................................skipped
When a step that we expect to be skipped....................................skipped
]]></system-out>
</testcase>
  1. The empty scenario and empty report are considered passing

To be consistent with the exit code of the Cucumber process empty scenarios
and test runs are considered passing.

⚡️ What's your motivation?

Partially resolve cucumber/common#871

🏷️ What kind of change is this?

  • ⚡ New feature (non-breaking change which adds new behaviour)

♻️ Anything particular you want feedback on?

📋 Checklist:


This text was originally generated from a template, then edited by hand. You can modify the template here.

@mpkorstanje mpkorstanje force-pushed the write-message-based-junit-formatter branch from f9e2bc7 to d3d73b8 Compare November 10, 2022 17:34
@codecov
Copy link

codecov bot commented Nov 10, 2022

Codecov Report

❗ No coverage uploaded for pull request base (main@38a4620). Click here to learn what that means.
The diff coverage is 76.47%.

❗ Current head 3432608 differs from pull request most recent head 8bd9818. Consider uploading reports for the commit 8bd9818 to get more accurate results

@@           Coverage Diff           @@
##             main    #2638   +/-   ##
=======================================
  Coverage        ?   84.68%           
  Complexity      ?     2680           
=======================================
  Files           ?      322           
  Lines           ?     9444           
  Branches        ?      899           
=======================================
  Hits            ?     7998           
  Misses          ?     1120           
  Partials        ?      326           
Impacted Files Coverage Δ
...n/java/io/cucumber/core/plugin/JUnitFormatter.java 73.33% <60.00%> (ø)
.../core/gherkin/messages/GherkinMessagesExample.java 80.00% <100.00%> (ø)
...core/gherkin/messages/GherkinMessagesExamples.java 80.00% <100.00%> (ø)
...erkin/messages/GherkinMessagesScenarioOutline.java 78.57% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@mpkorstanje mpkorstanje force-pushed the write-message-based-junit-formatter branch from 7c1c8d8 to 99c846f Compare December 3, 2022 15:08
@mpkorstanje mpkorstanje changed the title Implement a message based JUnit Formatter Use a message based JUnit XML Formatter Dec 5, 2022
[cucumber-junit-xml-formatter](https://github.com/cucumber/cucumber-junit-xml-formatter).
This achieves several goals:

 * Move the internal code base away from the events from the `plugin` module.
 * Extract common parts of Cucumber into modules that can be shared.

This results in several changes/improvements:

1. Change the long naming strategy to include the example group index

```feature
Feature: Examples Tables
  Scenario Outline: Eating cucumbers
    Given there are <start> cucumbers
    When I eat <eat> cucumbers
    Then I should have <left> cucumbers
    Examples: These are passing
      | start | eat | left |
      |    12 |   5 |    7 |
      |    20 |   5 |   15 |
    Examples: These are failing
      | start | eat | left |
      |    12 |  20 |    0 |
      |     0 |   1 |    0 |
```

This feature previously contained the examples
 - Examples Tables - Eating cucumbers - These are passing - Example #1
 - Examples Tables - Eating cucumbers - These are passing - Example #2
 - Examples Tables - Eating cucumbers - These are failing - Example #1
 - Examples Tables - Eating cucumbers - These are failing - Example #2

And will now contain the examples:
 - Examples Tables - Eating cucumbers - These are passing - Example #1.1
 - Examples Tables - Eating cucumbers - These are passing - Example #1.2
 - Examples Tables - Eating cucumbers - These are failing - Example #2.1
 - Examples Tables - Eating cucumbers - These are failing - Example #2.2

2. Rename the suite from `io.cucumber.core.plugin.JUnitFormatter` to `Cucumber`

```xml
<testsuite failures="0" name="Cucumber" skipped="0" errors="0" tests="1" time="0">
```

3. When a scenario is skipped or failed the executed steps are always included in the
   `system-out` element rather than the `skipped` or `failed` element.

```xml
<testcase classname="Examples Tables" name="eating cucumbers - These are failing - Example #2.1" time="0.007">
<failure type="AssertionError" message="Expected values to be strictly equal:

-8 !== 0
">
<![CDATA[
<a stacktrace here>
]]>
</failure>
<system-out><![CDATA[
Given there are 12 cucumbers................................................passed
When I eat 20 cucumbers.....................................................passed
Then I should have 0 cucumbers..............................................failed
]]></system-out>
</testcase>
<testcase classname="Skipping scenarios" name="Skipping from a step causes the rest of the scenario to be skipped" time="0.005">
<skipped/>
<system-out><![CDATA[
Given a step that skips.....................................................skipped
When a step that we expect to be skipped....................................skipped
]]></system-out>
</testcase>
```

4. The empty scenario and empty report are considered passing

To be consistent with the exit code of the Cucumber process empty scenarios
and test runs are considered passing.
@mpkorstanje mpkorstanje force-pushed the write-message-based-junit-formatter branch from 6263e16 to f451db8 Compare December 27, 2022 18:13
@mpkorstanje mpkorstanje marked this pull request as ready for review December 27, 2022 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Message based cucumber-junit-formatter
1 participant