Rules + Examples

[ryan-mars]

Summary

There is no useful mapping between Gherkin and "rules and examples".

Expected Behavior

I can express Rules and Examples naturally (aside from the fact that Gherkin is context free grammar and will never be as fully expressive as English).

Current Behavior

Features, Scenarios and Steps are the high level concepts. Rules and Examples must be shoe-horned in by the user.

Possible Solution

Feature: is replaced with Spec:

Rule: is a new section in-between what is currently Scenario: and Feature:

Scenario: is replaced with Example:

Scenario Outline: is replaced with Example Outline:

Spec: User sees the progress of processing for a new file

  Rule: Uploading accounts for 50% of processing 

    Example Outline: user sees upload at various states
       Given the user is uploading a new file
       When the upload is at <a_percent_uploaded>
       Then the progress bar should show <a_percent_complete>

       Examples:
       | a_percent_uploaded | a_percent_complete |
       | 10%                | 5%                 |
       | 25%                | 12.5%              |
       | 51%                | 25.5%              |
       | 75%                | 37.5%              |
       | 90%                | 45%                |
       | 99%                | 49.5%              |

  Rule: Preparing transcode job waits at 55%

    Example: A file is waiting on transcode job creation
      Given a file is waiting on transcode job creation
      When the progress bar updates
      Then we should see it is at 55%

  Rule: Transcoding happens from 60 - 75% and waits at 75%

    Example: File is transcoding
      Given a file is transcoding
      And it has been transcoding for <time> seconds
      When the progress bar updates
      Then it should show <percent> progress
      | time | percent |
      | 1    | 60      |
      | 2    | 65      |
      | 3    | 70      |
      | 4    | 75      |
      | 8    | 75      |

Context & Motivation

We teach that:

• The outcome of deliberate discovery should be rules and examples
• These rules and examples should become part of the team's living documentation
• We get living documentation by connecting examples to tests

The three most important concepts in Gherkin + Cucumber are features, scenarios, and steps.

There is no useful mapping between Gherkin and "rules and examples".

[aslakhellesoy]

I really like this. I'd love to be able to:

a) Generate and display an example map (in an example mapping tool) from existing Gherkin documents (that follow this new structure).
b) Generate skeleton Gherkin documents from an example mapping tool.

If we change the Gherkin grammar, it's essential that old documents still work:

• Example just becomes a new synonym for Scenario, which will still work
• Rule becomes a new keyword, which is optional

The AST would have to change so that every Gherkin Document has a list of Rule nodes, each of which has a list of Background/Example/Example Outline nodes. This would allow a Background per rule.

For legacy documents (no rules), we'd have to build the AST with a "dummy" rule.

Automatic rule validation

We've been wanting to add property based testing support to Gherkin and Cucumber for a while. Recently @ELetier introduced to a tool called Alloy, which can be used to validate rules using a different approach.

The overall idea is that Cucumber users would be able write Gherkin documents with rules without examples. A property based testing tool or a tool like Alloy (which would have to be integrated with Cucumber somehow) would generate examples at runtime, based on the properties associated with the rules.

I'm bringing this up because I want to make sure that a change to the Gherkin grammar to allow for rules also makes it possible to hook in property based testing tools and/or tools like Alloy.

For example:

Rule: An infant must sit in an adult's lap
Rule: An adult is 18 or older
Rule: An infant is 2 or younger

The text of the rules above would actually be interpreted (with a rule definition?), and there wouldn't (necessarily) be any explicit scenarios below.

Let's discuss!

[enkessler]

Those Rules don't look like they have a mechanical effect. Are they just a descriptive/organizational thing? Similar to RSpec's describe blocks? Could Rules have a mechanical effect such as being able to 'run' a rule (which would run all examples within the rule)?

[aslakhellesoy]

The way @ryanwmarsh suggested it initially, the only effect of Rule would be its presence in the AST, which other tools could use go generate an example map.

However, Cucumber (or a Cucumber plugin) could use the Rule to run a "rule definition", which could intergate with tools like property based testing tools or Alloy, to do validation of the rule itself.

[ryan-mars]

@enkessler I intended it to simply be present in the AST as @aslakhellesoy stated however, I recognize that some folks would like this to actually be a "Rule definition" as @aslakhellesoy also mentioned

[ryan-mars]

text of the rules above would actually be interpreted (with a rule definition?), and there wouldn't (necessarily) be any explicit scenarios below.
— @aslakhellesoy

So are we thinking the step def side would look something like:

// Given/When/Then
StepDef("An adult is traveling with an infant", function() {
    // implement this step
}); 

Rule("An adult is 18 or older", function() {
    // implement this rule
});

Or does the rule need some added context?

[aslakhellesoy]

Something like that. Not sure if the ruledef needs additional context, I'm not yet familiar enough with the other tools I mentioned.

[mattwynne]

Didn't @sebrose suggest something similar, maybe in the Gherkin repo? On phone now so can't easily search.

[aslakhellesoy]

Hello @cucumber/committers - I've written up some thoughts about how we could add property based testing to Cucumber over at cucumber-examples/shouty.rb#2.

Do you see any problems with this?

[gasparnagy]

I have no problem with the structure of the feature file, although I would like to see also what properties do you check of the rule in the Gherkin file itself.

In Cukenhagen we discussed about this and based on that discussion I made this example:

https://github.com/gasparnagy/SpecFlow.FsCheck/blob/master/sample/AdditionSample/MyCalculator.Tests/Addition.feature

But of couse this can be an optional part. If there would be a "Rule" level in the feature file and some way to hook any kind of automation for it that's a good start...

[mgundala]

I would like to know the difference between "Rule" and "Scenario", aren't we conveying the same in both the keywords?

According to my knowledge "Rule" is a condition that needs to verify through Gherkin Steps.
The scenario is a test Case that also needs to verify through Gherkin Steps.

Is Rule is a kind of Condition check for Scenario?

[aslakhellesoy]

@mgundala see @mattwynne's article about Example Mapping for an explanation of rules vs scenarios (aka examples). This is the technique we recommend for designing scenarios in three amigos sessions.

[aslakhellesoy]

@ryanwmarsh what's your motivation for suggesting to rename Feature to Spec?

[mattwynne]

IMO we don't need Example Outline and I don't like how it reads. I've always thought of a scenario outline as a rule, with the examples in the table below being illustrations of that rule.

So ideally for me we'd have:

Spec: User sees the progress of processing for a new file

  Rule: Uploading accounts for 50% of processing 
    Given the user is uploading a new file
    When the upload is at <a_percent_uploaded>
    Then the progress bar should show <a_percent_complete>

    Examples:
    | a_percent_uploaded | a_percent_complete |
    | 10%                | 5%                 |
    | 25%                | 12.5%              |
    | 51%                | 25.5%              |
    | 75%                | 37.5%              |
    | 90%                | 45%                |
    | 99%                | 49.5%              |

  Rule: Preparing transcode job waits at 55%

    Example: A file is waiting on transcode job creation
      Given a file is waiting on transcode job creation
      When the progress bar updates
      Then we should see it is at 55%

i.e. you can use Rule either as just a container of regular Scenarios, or as a scenario outline with table-based examples.

[mattwynne]

@lunivore I'd love to know what you think about this

[SabotageAndi]

I was thinking the last days about this, but I am still not sure about this suggestion.
How fix are the new keywords?
I have the fear, that the reuse and combination of some keywords (example, outline) will irritate people, especially people that start using Gherkin.
Also I am currently not a fan of the shortening of Specification to Spec. In English this works to a degree, but the short form of "Spezifikation" in German would be Spez or Spek. This doesn't look good in a professional environment.

[aslakhellesoy]

As there are several suggestions in this issue I suggest we focus on adding Rule and ignore the following suggestions:

• Rename Feature to Spec
• Rename Scenario to Example
• Rename Scenario Outline to Example Outline

Feel free to create separate issues for those. This issue is exclusively about adding a Rule keyword from here on.

[enkessler]

Maybe it's just because I'm not familiar either rules and property based testing, but every time I see Rule in a feature file, my mind tries to interpret it as a setup step that only applies to scenarios within its scope.

Rule('Shouts have a range of {int}m') do |range|
  Shouty.set_range(range)
end

But it looks like a rule definition is no different from a step definition so I suppose that different people could use it in different ways.

[brasmusson]

@enkessler Apart from the Example Mapping article, I think the @lunivore blog post Acceptance Criteria vs. Scenarios is a good description of the meaning of "Rule" (defined by the Acceptance Criteria) and "Example" (defined by the Scenarios) as used in the discussion.

[lunivore]

Hey Matt, Scenario and Example are pretty much synonyms, with the first being easier to imagine than the second. Historically we've used Example at a unit / class level, and Scenario at a system level. I'm happy for that to continue. "Give me an example" and "Give me a scenario where that happens" are both good questions to get the conversation started. If you want a really good word it's "Exemplar" but hardly anyone knows what that means. I prefer to stick with plain English where possible. Please don't remove "Scenario"; if you want to support "Example" as well, that would be good. I would really love to avoid "Specification" just because it makes it sound as if things are concrete. I find those words tend to make people skip the "exploration by example" bit of BDD, from which the specs and rules should be derived. I prefer "Rule" to "Specification" or "Acceptance Criteria" (which makes people think of software instead of behaviour). At least we know rules are made to be broken. (Also "Acceptance Criteria" is plural; it's "Criterion" if you've only got one.) With respect to "Shouts have a range of {int}", that's a rule / criterion. If I then ask, "Can you give me an example?" you have to think of "The one where...", hence, "The one where the queen shouts for Charles for dinner and he's in Trafalgar Square" (we actually did this in one of the courses that Aslak and I took jointly). That's an example. Or a scenario. This becomes a lot clearer when there are different contexts in play, so if in doubt, look for examples that have those. I really like Shouty as an exercise though because it's new enough that people can actually practice the exploration bit. It's really hard to explore things that aren't new. Re Feature: meh, I don't really mind. I use capabilities, but sometimes we need to subdivide capabilities by context or even by outcome when they become extensive. Features are implementations of capabilities in contexts. Consider using "Theme" if you want something non-software-related. That works for everything, including elephants. Also please ping me on Twitter or something if you want my thoughts; really happy to give it but I only saw this by chance! Cheers, Liz.

…

On Thu, Oct 5, 2017 at 9:57 PM, Matt Wynne ***@***.***> wrote: @lunivore <https://github.com/lunivore> I'd love to know what you think about this — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#250 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA_ZkNhjsiKnscwieN46RsIzEGdCBTJHks5spULLgaJpZM4O80LW> .

-- Liz Keogh liz@lunivore.com http://lizkeogh.com http://lunivore.com

[brasmusson]

@mattwynne I do not think a Scenario Outline is a Rule, but a Scenario Outline can be used to define the set of Examples illustrating a Rule. I think a statement like "the Steps of a Rule" is nonsense, therefore I think it would be a mistake to place the Outline Steps directly below the Rule line, there need to be something in between to mark the line between the Rule itself and the Examples illustrating the Rule.

[dotnetprofessional]

I'm still a little confused over the addition. Is the idea to provide an intermediate level of description for the feature? That is I find feature is at a level down from what I might want, in so far as an Epic would describe a larger piece of functionality and the features would break that down, with the scenarios being more concrete user interactions. This 'seems' to be similar in that a rule is a level down or a way to group a set of scenarios. In our Features we often put as part of the description a bullet point list of Rules which are trying to give a high level description of the scenarios below. So if we have the additional Rule keyword it would wrap the scenarios/examples for that rule. I can see value in that.

I'm still not clear why you'd add validation to the rule however, if the scenarios/examples are filling that role. If you want to make it more terse then having separate keywords for Specification that then have multiple When, Then, When thens etc. would seem clearer.

[mlvandijk]

A 'Rule' would specify a particular business rule (i.e. the range in Shouty, or the age at which Fluffy bunny can be sold); traditionally called 'acceptance criterion' in software development. Imho it would be helpful to explicitly specify these rules, as they would be tested by the scenarios/examples but might not be explicitly mentioned here.
To me a Scenario or Scenario Outline specifies a collection of steps to be taken (Given/When/Then); the examples that go with a scenario outline are different sets of input -> expected output.

I do not understand if/how a Rule would then also be implemented to actually 'do' something (as in the example PR by @aslakhellesoy ); to me it would be additional background info to state which Rule (or Rules?) are being tested in certain Scenarios/Examples. The Scenarios/Examples would specify how the 'Rule" is applied in different situations and test that the 'Rule' is correctly implemented in the system.

However, it could also be used as a 'setting' (as the range in the Shouty example), so you could test for different versions of a Rule? (i.e. different ranges). If that is the case, then I would wonder how this is different from a Given or a Background?

[lunivore]

Criteri on, not criteri um (it's Latinized Greek -> κριτήριον ), plural criteria.

Rules might be about context (Given / Background), but they might also be triggered by actions or only affect outcome. So for instance:

• Given a trade contract has been generated, when it takes more than 2 days to audit, then the system should send a reminder <- triggered by action of time passing.

• Given a trade for chocolate has been scheduled over Christmas, when we request the contract for the trade, then a note that it's a Christmas trade should appear at the top of the contract <- we already generate contracts; this is just another rule that only affects the outcome.

[mlvandijk]

Now I'm possibly more confused about the use of the 'Rule' keyword.
To try and clarify, I'd rewrite:

Rule: Trade contract should be audited within 2 days.
Scenario:
Given a trade contract has been generated
When it takes more than 2 days to audit
Then the system should send a reminder

So, there's a 'Rule'; it is exemplified(?; I mean that an example is provided) / tested by the Scenario, the specific rule may come into play at any step in the scenario though?

Second example: what does 'over Christmas' mean?

[lunivore]

Some rules require n+1 examples to illustrate. So in this case, you'd have two: the one where it took more than one day to audit, and the one where it didn't.

I only illustrated one because that wasn't a facet I was focused on, but there it is.

If you have a lot of different rules, you often have the "happy path" (your +1) and then all the edge cases ("n"). If you're just adding a new outcome, you're introducing a new rule without introducing any new examples, but you'll be extending an existing one. I'm sure there's some kind of mathematical construct, but my main point is that rules can have multiple examples.

Remember though that the main purpose of BDD is to capture conversations, which explore the space and only produce specifications and tests as a nice by-product.

We just had an exercise today for which we had a post-it on our forehead and could only ask yes/no questions in the room, then had to find our matching post-it pair. My question was, "Can you see my pair from here? Is my pair in that direction?" Perfectly within the rules, but I'm pretty sure it wasn't what they intended. That's the kind of thing that examples help to clarify.

[gasparnagy]

Do we have a conclusion here?

Considering the idea of adding an additional, but optional "Rule" grouping around scenarios and scenario outlines...

• As it seems we can all be fine with this (even though some of us are not fully sure about the benefits).
• Since this would be an optional grouping, this would still allow situations where this concept does not fit well or when the team needs more time to adopt.
• I think we all came to the conclusion that neither Scenario Outlines nor the Feature file itself is a good replacement for rules (even with a different aliases) because we 1) sometimes want to collect examples of different text structure under a rule that would not be possible with one SO 2) we want to support simple rules with 1-2 scenarios, where a new file for each rule would be inconvenient.
• It is a little bit unclear how we can attach direct test logic to rules (e.g. with property-based testing), but this is anyway more related to the actual tool/platform and not so much on Gherkin anyway, so maybe we could still keep this part experimental.
• Renaming the other artifacts (Scenario->Example, Scenario Outline->Example Outline, etc. can be discussed separately, as @aslakhellesoy suggested)

I think the introduction of a Rule level grouping is quite an safe change. Even if the change will not be backward compatible it is easy to make an extension that provides the "flat" scenario structure for a feature file by visiting the rule/scenario tree.

I run into this problem (lack of structure to express rules) more and more often and I would really like to have better support for it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[ryan-mars]

Wow great discussion here. Somehow I didn't get the notifications. So many great points. I apologize for letting this go stale. 🙏

• I think @mattwynne's example is spot on. 👍

• I acknowledge @lunivore's concern about people avoiding exploration by example because specification is a strong word. The problem there would be culture. A given team can collaborate on a specification or it can follow anti-patterns. Spec is a well known term (RSpec, SpecFlow, etc...). IMHO if we have a word that accurately describes a thing then we should call it by that word. Fixing culture on a team is a separate problem. However, I am always open to a better term. 🤔

Now allow me to get crazy for a moment. I'm boiling the 🐸 slowly here.

What if rules and examples didn't have to be in the same file?

Spec: User sees the progress of processing for a new file

  Rule: Uploading accounts for 50% of processing 

  Rule: Preparing transcode job waits at 55%

  Rule: ... 
     ... free text description 

  Rule: ... 
     ... 

  Rule: ... 
     ... 

Spec: Examples for uploading

  @rule:uploading_accounts_for_50_of_processing
  Example: 
    Given the user is uploading a new file
    When the upload is at <a_percent_uploaded>
    Then the progress bar should show <a_percent_complete>

  Examples:
    | a_percent_uploaded | a_percent_complete |
    | 10%                | 5%                 |
    | 25%                | 12.5%              |
    | 51%                | 25.5%              |
    | 75%                | 37.5%              |
    | 90%                | 45%                |
    | 99%                | 49.5%              |

  @rule:preparing_transcode job_waits_at_55
  Example: A file is waiting on transcode job creation
    Given a file is waiting on transcode job creation
    When the progress bar updates
    Then we should see it is at 55%

  @rule1 @rule2 
  Example: 
    Given ..
    When ..
    Then .. 

Here's why. In some organizations rules are a meaningful artifact. In some organizations rules must be given an identifier so they may be referenced in various forms of documentation and discussion. Alternatively rules must correlate with an industry regulation. What I have proposed is an incomplete solution for organizing and referring to rules. How do we reason about the rules in our system when we have hundreds?

My suggestion:

• Make rules and examples first class citizens.
• Allow examples to refer back to rules, lets call this a rule-ref
• rule-refs should be human readable

Boiling the 🐸 further.

We still have the problem of terms. In many regulated businesses the terms used in rules matter greatly. Where are these terms to be defined? My gut says terms need to be first class citizens as well. I feel I might be considered disruptive with this but that is not my intention. I care that we say Cucumber is how we connect tests to rules and examples to produce "living documentation".

¯\_(ツ)_/¯

[mattwynne]

I agree about being able to relate Rules and Examples across Gherkin docs. I don’t see the need for a rule-ref, or the additional syntax to go with it. I think it would be in the spirit of Gherkin’s simplicity to just use the name of the Rule as it’s unique ID. So by duplicating the name of the rule in several places you could associate more scenarios to the Rule.

I think it’s time for someone here to make a concrete proposal and we can go ahead and implement something.

[jon-acker]

@ryanwmarsh I'm very much in favour of incorporating the Rules keyword, I almost always include it anyway. I'd be wary of separating them into two files though. Without capability to manage the relationship between them (would Cucumber do this?) we would quickly end up with unrelated rules and examples or rules without examples, etc.
I agree with @lunivore on keeping the Scenario keyword (I might go for example as I use it interchangeably), and I'd leave Feature as it's the most familiar way of grouping system functionality.

[aslakhellesoy]

This has been done on master and will be released in Gherkin 6. Then we need to wait for Cucumbers to upgrade to Gherkin 6, which is quite a bit of work.

But it's happening at least!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.