Consistency of Voight Kampff Steps

[codyreinold]

Issue / Inconsistency

Currently, the structure / verbiage of Voight Kampff steps appears to be a bit inconsistent. I noticed this as I was looking into implementing a few improvements re: language and skill settings.

The rhetoric is varied, as is the order of the phrasing (subject, verbs, descriptions, etc.). This is true between step types (given vs when vs then), but also true within a given type of step.

Given the opportunity for this testing suite to expand and be a common interface between the Mycroft project and community developers, I think that developing a consistent standard re: step phasing structure would be helpful.

Proposed Solution

Given Steps

As I think about the Given steps and future uses, two specific uses come to mind.

• Setting global settings (eg: language, timezone, device location)
• Setting skill-specific settings

This section seems to be largely "setting the scene", ie: setting variables. As such, I would recommend we follow a familiar structure: the phase equivalent of x = y (square brackets added for clarity of pattern; would not be in actual test structure).

• "Given the user's [language] is [English["
• "Given the user's [location] is [France]"
• "Given the skill's [SETTING_NAME] is [SETTING_VALUE]"

I think this structure provides a very consistent template / pattern for developers to follow, and allows us to easily expand Given testing options in the future while following the same general structure.

When

When steps already follow a similar pattern to Given, using "the user", not "a / an user" and leading with the subject of the phrase. No changes necessary.

• "When the user says [PHRASE]"

Then

The final type of step, Then steps, also provide an opportunity for standardization. Given Then steps are (currently) largely text / dialog-related, stndardization could follow a relatively simple pattern:

• Then
• Subject: ['mycroft', '[SKILL_NAME]']
• Action / Conditional: ['should reply with']
• Indirect Subject: ['[EXACT_STRING]', 'dialog from file [FILE]', 'anything', 'nothing']

Finally, I believe the last block in the codebase, dedicated to continued user interaction with the system (eg: "Then the user replies with [STRING]"), should be recategorized to a When step type. This aligns with the Behave documentation:

Given we put the system in a known state before the user (or external system) starts interacting with the system (in the When steps). Avoid talking about user interaction in givens.

When we take key actions the user (or external system) performs. This is the interaction with your system which should (or perhaps should not) cause some state to change.

Then we observe outcomes.

As this is an additional interaction, it deserves the When step type. I think this can be concisely communicated as When = user interaction; Then = system response. This will allow us to streamline the codebase and

Action

I am happy to make these changes in the underlying codebase / provide documentation to support it. Wanted to get the opinion of others on this topic, though.

[forslund]

I'll tag @dschweppe who is currently looking over the wording of the feature files.

You might also want to chime in on the discussion in #2547 where configs are discussed a bit.

[codyreinold]

Yes, that issue is part of what caught my initial attention. I'll throw a comment in over there. I will also pass along some code in a bit that will show what the implementation could look like.

[krisgesling]

First off, I wanted to say thank you for such a clear and well articulated description.

Because it's easy to add variations on the language, it's quite tempting to do so, but having a clear and standard structure will certainly make it easier. Even just reading longer feature files will be easier when your brain can recognize the patterns and more easily skim through.

As the When steps expand, it would be good for these to follow the same type of format:

• When
• Actor / subject: the [user, Skill, external service, ...]
• Action / verb: [says, asks, loads, pushes new data, ...]
• Phrase / indirect subject (optional): [utterance, http response code, ...]

Also agree on changing Then > When in "Then the user replies with [STRING]"
Just double checked and there doesn't seem to be a restriction on Step flow, eg you can have a Scenario with the following Steps:

Given ...
When ...
Then ...
When ...
Then ...

[forslund]

The use of Then instead of When must have been a mistake on my part when doing the implementation, when I tried back then I got an error having a when after a then but it seems to be working, so I must have been mistaken.

The only benefit of having it as a then step is that the And syntax can be used to make the flow more natural.

And the user replies...

[forslund]

If you want to start on parts of this I think we can all agree on the change of the Then the user replies to "When the user replies" (we should keep the then version as a deprecated backwards compatibility to be removed in 20.08) as well as adding the suggested configuration related givens. I did start on the config for system unit selection earlier this week and will try to finish that.

I also think your suggestions about how to keep a consistent structure makes sense, but I'll leave the decision on exact terminology in the hands of @krisgesling and @dschweppe since they
A: unlike me are native English speakers
and B: has put some thought into the wording.

[dschweppe]

These are great suggestions. I agree that using When for the user input makes the most sense. We are still discussing what we want to be the suggested method for writing Then steps (Mycroft responses). We should support ['[EXACT_STRING]', 'dialog from file [FILE]', 'anything', 'nothing'] but right now we will encourage people to use 'dialog from file [FILE]'. However, we are thinking of ways to support writing Mycroft responses in a more natural way.

[forslund]

Closing Issue since we're archiving the repo