Proposal: --wip-tags TAG_EXPRESSION

[aslakhellesoy]

This is a proposal to allow for "work in progress" scenarios to fail.

Context: Occasionally, a team will work on a scenario that depends on other functionality (possibly out of their control) to be implemented before the scenario can pass. For example, a UI needs some new data from a backend. The UI work is completed, but the backend hasn't yet been updated to provide the data.

In this context, the scenario could be marked with a tag (for example @wip for "work in progress") and run with --wip-tags @wip.

This would allow developers to commit the half-finished scenario and allow the build to stay green. As soon as the scenario fails (when the missing functionality is implemented), the build would fail. This is an indication that the @wip tag should be removed from the (now completed) scenario.

It should be noted that some Cucumber implementations already have a --wip option, which is similar. The drawback of this option is that it requires a separate invocation of Cucumber. If this proposal is accepted, we should deprecate (and ultimately remove) the --wip option.

[mpkorstanje]

If this proposal is accepted, we should deprecate (and ultimately remove) the --wip option.

I don't quite follow this. It would appear to me that --wip and --wip-tags are different work flows.

[aslakhellesoy]

The way I see it, the current --wip option is an inferior way to do the same as what you can do with --wip-tags.

With just --wip at my disposal I need to invoke Cucumber twice:

• --tags "not @unfinished"
• --wip --tags "@unfinished"

With --wip-tags, a single invocation is sufficient:

• --wip-tags "@unfinished"

[mpkorstanje]

Could we stream line this to --wip "@Unfinished" and have --wip imply that everthing is unfinished?

[ciaranmcnulty]

I'm not sure WIP features should have to fail - for instance they may be flakey. perhaps a second category of result / some sort of warning would be better?

Essentially, something unexpectedly working shouldn't be a suite failure to me.

[aslakhellesoy]

Could we stream line this to --wip "@unfinished" and have --wip imply that everthing is unfinished?

Yes, I think that's a good idea.

I'm not sure WIP features should have to fail - for instance they may be flakey. perhaps a second category of result / some sort of warning would be better?

So when a WIP scenario suddenly starts passing, print a warning? Yes I think that might be better. Would this mean a new status for a scenario? wip-success? And then print scenarios with this status as yellow and still pass the build?

[ciaranmcnulty]

Would this mean a new status for a scenario? wip-success? And then print scenarios with this status as yellow and still pass the build?

Yes, something along those lines. I was picturing frustrated devs trying to merge their PRs and not being able to because 'it works more than it's meant to!'

[mpkorstanje]

Nobody reads build logs though. If the feature was implemented and now works but the wip tag not removed it is a failure to update the documentation.

[aslakhellesoy]

It could fail by default and pass if —no-strict is specified.

[mpkorstanje]

cucumber/cucumber-jvm#1381

The interaction between wip and strict is already complicated. :D

[aslakhellesoy]

I’m sure it can be simplified

[aslakhellesoy]

Could we stream line this to --wip "@unfinished" and have --wip imply that everthing is unfinished?

On second thought I don't think this is possible. Consider the following argument lists:

--wip @foo
--wip features/foo.feature

For the first one it's clear that @foo is an argument to --wip. For the second it's clear that features/foo.feature is not an argument to --wip.

For the optparse code however, building this knowledge in is tricky. It would have to interpret the 2nd argument and decide whether it's a tag expression or a file, and I don't think an option parser should have that kind of knowledge.

For this reason, I don't think options should have optional arguments. We have the following alternatives:

• Change --wip to expect a mandatory argument (this would break backwards compatibility and lead to some confusion)
• Introduce a new --wip-tags option with a mandatory argument and deprecate --wip.

I think the latter is better.

[mpkorstanje]

I don't think an option parser should have that kind of knowledge.

It should. Arguments shouldn't travel beyond the option parser. But it also wouldn't be able to determine what --wip @rerun.txt means so the point is moot.

Introduce a new --wip-tags option with a mandatory argument and deprecate --wip.

Agreed but I am not sure about the deprecation. --wip --tags @wip only executes features tagged as @wip. While --wip-tags @wip runs everything. I guess --wip-tags @wip --tags @wip would be the replacement for that.

[amp4045]

I kind of see a true "work in progress" may mean that I'm writing steps within my feature file, but I may have undefined step definitions. Would a wip-tag also be the appropriate tool to effectively "skip" the entire scenario? Perhaps I'd still like knowledge of the scenario's existence within my report.. as if it were a "TODO" in code.

I say this as in my experience, we have occasions where someone else is writing the feature file than the person implementing the step definitions. We implement the feature file (which has undefined steps) as a '.future' file such that it can be baselined, but for obvious reasons would not execute. It'd be interesting to explore the idea of having a WIP tag that'll "print" the scenario to the report but not impact any test evaluation (build errors, etc.). This could be very effective when coupling cucumber with other tools such as cucumber reporters for Jenkins as it creates an individual report for all tags analyzed while running cucumber. (In my mind you'd then have a collection of scenarios with "TODOs (WIPS)" and maybe that's okay) Thoughts?

[mpkorstanje]

I think that use case is already partially covered by the PENDING state (achieved by throwing a pending exception). Depending on the --strict and --wip settings that may fail the build.

Currently the truth table of scenario result x strict x wip looks like:

           |       --wip            |       --no-wip         |
           | --strict | --no-strict | --strict | --no-strict | 
PASSED     |   fail   |  fail       |   pass   |  pass       |
SKIPPED    |   pass   |  pass       |   pass   |  pass       |
PENDING    |   pass   |  pass       |   fail   |  pass       |
UNDEFINED  |   pass   |  pass       |   fail   |  pass       |
AMBIGUOUS  |   pass   |  pass       |   fail   |  fail       |
FAILED     |   pass   |  pass       |   fail   |  fail       |

It's up to the implementation of a reporter how they display each scenario.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[stale]

This issue has been automatically closed because of inactivity. You can support the Cucumber core team on opencollective.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.