HTML Report includes all features, not listening to tag filtering.

[dnldpavlik]

Describe the bug
When using the HTML Formatter for --format, I get an HTML document that contains all features, even ones that were not executed during the test run. Each feature is in a collapsable container, also there is no header with the run stats.

To Reproduce
Steps to reproduce the behavior:

1. execute cucumber -f HTML --out report.html -t 'not @exclude'
2. Open the report and see @exclude features are listed in the report.

Expected behavior
HTML report to be generated with only features that were executed.

Context & Motivation

This is breaking the report runner that I use to aggregate several cucumber test runs into a summary report.

Your Environment

• Versions 4.1.0
• Windows Serve 2016 Datacenter

[vincent-psarga]

Hi and thanks for the report,

Both issues should be fixed by cucumber#1111
I thought those issues were solved but the reporter did not use the correct component for rendering.

That said, the not executed feature will still be in the HTMl report, but hidden by default. Would that be enough for you ?

[luke-hill]

@vincent-psarga One question about the above. Would this only be for features that are tagged but excluded? Or would this be for all different "types" of permutations.

i.e. Features in certain directory (Would the other features not ran be in the report hidden)
i.e. Features that match an inclusion tag, such as -t @positive (Would all other features without this tag / no tags, be in the report hidden)

Reason I ask is some companies would have a features directory with say 500 features, but only ever run 20 per execution. So if the report is creating data for 500 features, it would be a lot of excess work?

[vincent-psarga]

That's a good question :) The features/scenarios will be in the HTML report if they have been parsed by Cucumber.
Basically, it will depend on the way you run the suite. I'll take the example of cucumber-ruby (and considering I added the @excluded tag on features/docs/cli/dry_run.feature).

bundle exec cucumber -> 81 GherkinDocument messages emitted, so all feature would be in the HTML report
bundle exec cucumber -t "not @excluded" -> same messages, so all features in the HTML report
bundle exec cucumber features/docs/cli -> 16 GherkinDocument messages emitted (so the ones that are not even "loaded" (or whatever the correct term is) will be displayed

I don't think it'd be a huge investment to really remove the not executed feature in the HTML report, it's a use case that we did not think of before.

[AlphaInfamous]

is there a timeline on when this issue can be fixed.

[vincent-psarga]

is there a timeline on when this issue can be fixed.

the PR (cucumber#1111) about the html-formatter has been merged, so hopefully I'll be able to cut a release for those.

[AlphaInfamous]

@vincent-psarga Thank you. waiting for the release.

[AlphaInfamous]

Hi Updated to Cucumber 5.0.0 and now all my html reports are being generated as Blank. (JSON is not empty.)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[stale]

This issue has been automatically closed because of inactivity. You can support the Cucumber core team on opencollective.