Execution Summary details at top right corner

[jega-kathir]

Hi
I ran and created one sample html report.
I ran for 2 features which has 11 scenarios.
In this, 4 scenarios are failed (2 from Feature 1 and 2 from feature 2)
But report says:

Features: failed: 2
Scenarios: failed: 2
Steps: untested: 8

Here am not able to understand how the "Scenario : failed" and "Steps : Untested" got calculated.

[bittner]

Do you have some code we can look at to better understand the problem?

What is the console output of your tests when you don't use the HTML formatter?

[jega-kathir]

My console Output:
0 features passed, 2 failed, 0 skipped
7 scenarios passed, 4 failed, 0 skipped
40 steps passed, 4 failed, 0 skipped, 0 undefined
Took 0m36.643s

But in report
html error.docx

Features: failed: 2
Scenarios: failed: 2
Steps: untested: 8
Finished in 36.6 seconds

[jenisys]

@jega-kathir
The untested state is the initial state before the test run.
Therefore, if you see untested steps this means that these steps did not get executed during the test run.

EXAMPLES:

• If the second step in a scenario fails the remaining steps in this scenario have still the state untested (because the execution normally stops after the first failed step in a scenario).
• If you are using dry-run mode, all steps are untested (unless you have undefined steps that have no step implementation) because they do not get executed during the test run.
• If you want to stop the test run after the first failure, the other scenarios, that would normally be executed, are untested.

OTHERWISE:
Your console output example above is weird.
untested steps are optional in the summary. Therefore, untested steps are only shown if some exist.

[lukjak]

I have a similar issue.

Behave output states:

0 features passed, 3 failed, 0 skipped
5 scenarios passed, 7 failed, 0 skipped
11 steps passed, 7 failed, 0 skipped, 0 undefined
Took 9m40.783s

And HTML report (with v 0.9.10) shows:

Features: failed: 3
Scenarios: failed: 4, passed: 1
Steps: passed: 3, failed: 3, untested: 3
Finished in 580.8 seconds

[lukjak]

Re my previous comment.
Feature data are OK.
Scenario data value accumulate all scenario outline executions into single item, while behave treats each execution as a separate scenario.
I have no explanation for steps value - it seems to be completely wrong.

Here is results outline:

Feature 1
Scenario 11
Step A - pass
Step B - fail
Scenario 12
Step A - pass
Step C - fail

Feature 2
Scenario Outline 21@1
Step D - pass
Step E - pass
Scenario Outline 21@2
Step D - pass
Step E - pass
Scenario Outline 21@3
Step D - pass
Step E - pass
Scenario Outline 22@1
Step F - pass
Scenario Outline 22@2
Step F - pass
Scenario Outline 22@3
Step F - fail
Scenario Outline 22@4
Step F - fail
Scenario Outline 22@5
Step F - fail
Scenario Outline 22@6
Step F - fail


Feature 3
Scenario 31
Step G - pass
Step H - fail