Feature/Cucumber Messages

[clrudolphi]

Adds support for creation of Cucumber Messages during the execution of Reqnroll generated tests.
See Cucumber Message for information about the output format.

🤔 What's changed?

Changes are in three main area:

1. Feature Generation: during feature-file code-behind code generation, the Gherkin toolchain is used to create the first few Messages for the feature. Factory methods are generated for each of these message types. These are attached to the FeatureInfo object.
2. Cucumber Message Publication: written as an embedded plugin, this component listens for Reqnroll ExecutionEvents and converts them, and their attached state, to Cucumber Messages. These are 'published' to one or more consumers.
3. FileSinkPlugin: an embedded runtime plugin, this plugin registers with the Publishing component to receive the Messages as they are released by the Publisher and stored to a file.

Messages are strings serialized from the CucumberMessage Envelope format and stored one per line in an ".ndjson" result file.

Configuration support is added as a new section of the reqnroll.json configuration file. (Without interfering with the existing configuration schema and config reading mechanisms). Configuration settings can also be overridden by use of environment variables.

⚡️ What's your motivation?

To provide future compatibility with the broader Cucumber eco-system of tooling.
To provide the input to a future implementation of a Reqnroll replacement to LivingDocs.

🏷️ What kind of change is this?

• ⚡ New feature (non-breaking change which adds new behaviour)

♻️ Anything particular you want feedback on?

Anything.

📋 Checklist:

• I've changed the behaviour of the code
  • I have added/updated tests to cover my changes.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly.
• Users should know about my change
  • I have added an entry to the "[vNext]" section of the CHANGELOG, linking to this pull request & included my GitHub handle to the release contributors list.

[clrudolphi]

FYI, github Build is failing b/c one produced component (FileSinkPlugin) gets published only to my local Nuget store. It can then not be found by the build pipeline in github.

[clrudolphi]

First end-to-end working scenario is operational.
Needed out of a PR review:

• Overall code structure
• Patterns that have been used (good/bad) and patterns that should have been used
• Thread-safety: is the level of thread awareness and approach to thread-safety appropriate and adequate?

Next:

• Copy the Cucumber Compatibility Kit scenarios into an integration test suite and build out the structure to build, invoke, and confirm compliance.

[clrudolphi]

Build is failing b/c I have a private build of Cucumber Messages nuget package. When the next release of that comes out, I will patch up the nuget references in this branch.

[gasparnagy]

Build is failing b/c I have a private build of Cucumber Messages nuget package. When the next release of that comes out, I will patch up the nuget references in this branch.

This is fine. We have feed https://www.myget.org/feed/Packages/reqnroll-unstable where we could publish interim releases (of dependencies as well) and I think this feed is added to the nuget sources, so the build would be able to resolve it though them. But I never tested this model yet.

[clrudolphi]

Please also note, that if someone uses a cucumber expression like foo {string} bar to match to foo "hello" bar this will set the position to the letter h of the hello, not the " before.

True. I've confirmed this by looking at the html generated from the CCK. {An aside, the Attachments scenario in the CCK is the only example of using {string} cucumber expressions. Wild that they don't have a scenario just for expressions.} The CCK html has the quotation marks in blue, while the html that is generated from our Messages does not.

How much of a deal-breaker is that?

[gasparnagy]

How much of a deal-breaker is that?

Not at all if this is only about the coloring and there is no duplicated displaying of the quotes.

We could use match.Index instead of g.Index which points to the quote char, but then we would also need to record the length as well (from match.Length) because it is longer than the actual value length (I don't know we need the length at all). But just for the color I would not do that. I think it is even prettier is the quotes are not colored.

[clrudolphi]

But just for the color I would not do that.

OK. I consider this question resolved.

[clrudolphi]

TODO: in side-bar discussion with @gasparnagy it was agreed to switch back to UUID id generation and remove the run-time rewrite of static message IDs. This is to avoid the performance and memory pressure overhead of rewriting sequential IDs.

Completed with 15f0230

[clrudolphi]

TODO: there is a bug with the sequences of messages when Attachments are published (they are being published before the TestCaseStarted event).

RESOLVED with 71aac1b

[clrudolphi]

Commit 15f0230 moved ID Generation to UUIDs.
This commit has broken the validations executed against some of the CCK scenarios. Still investigating why.

Resolved with commit aaf9d06

[clrudolphi]

Feature-level Tag handling: there is a discrepancy between the Gherkin PickleCompiler (for .NET) and the Cucumber Compatibility Kit.
The CCK has an example for Retry in which a @retry(3) tag is placed on the Feature (but not on the individual Scenarios). The CCK indicates that the tag should only be present in the Source and GherkinDocument messages.

The .NET Gherkin PickleCompiler copies those tags down to the Scenario level and thus they get emitted as tags on the Pickles that get generated. See this line of code in the PickleCompiler.cs

Does this behavior match what Reqnroll does internally during its handling of code gen from the AST?
If the PickleCompiler is wrong, I'll submit a PR to remove that line. If it is correct, I'll contact the CCK team to ask for clarification and perhaps get the CCK generated samples corrected.

[gasparnagy]

Does this behavior match what Reqnroll does internally during its handling of code gen from the AST?
If the PickleCompiler is wrong, I'll submit a PR to remove that line. If it is correct, I'll contact the CCK team to ask for clarification and perhaps get the CCK generated samples corrected.

Logically the feature tags are flowing down to the scenarios, but for some other reasons, we document them separately in the ScenarioInfo. This is the generated code of a scenario: both feature and scenario tags are applied to the method, but in the
constructor of the ScenarioInfo they are passed in separately.

[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("feature-tag")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("sceanrio-tag")]
public async System.Threading.Tasks.Task AddTwoNumbers()
{
    string[] tagsOfScenario = new string[] { "sceanrio-tag"};
    //...
    var scenarioInfo = new global::Reqnroll.ScenarioInfo("Add two numbers", null, tagsOfScenario, argumentsOfScenario, featureTags);

We currently build the code from the original AST and not from the pickles, so it does not really matter how the pickle compiler works currently. So to check the correctness of PickleCompiler I think the only chance is to compare it with the Java or Javascript versions where this is heavily used. My guess is that it's rather the CCK what is wrong, but not sure.

[clrudolphi]

@gasparnagy build is failing while building docs. I had included two pages I had written on Messages. The TOC references them but the build can't find them? Not sure why.

[clrudolphi]

Fix: stack traces were left off of StepFinished messages. Fixed with 61c7057

[clrudolphi]

Fixed issues that were preventing NUnitRetry scenario from validating. These were issues in the validation suite and should make test validation more reliable.
Commit 84de6ec

[clrudolphi]

I had asked the Messages team about adding message types for Before/After Feature hooks. It would seem that they've decided No on that for now (see.

Reqnroll Messages simply maps Before/After Feature hooks into TestRunHookStarted/Finished messages.
Is that acceptable?

[gasparnagy]

build is failing while building docs. I had included two pages I had written on Messages. The TOC references them but the build can't find them? Not sure why.

I will check and fix it. Please ignore for now.

[gasparnagy]

Reqnroll Messages simply maps Before/After Feature hooks into TestRunHookStarted/Finished messages.
Is that acceptable?

Yes, for now. But I will make a comment about it in the cucumber messages list.

[clrudolphi]

Feature-level Tag handling: there is a discrepancy between the Gherkin PickleCompiler (for .NET) and the Cucumber Compatibility Kit. The CCK has an example for Retry in which a @retry(3) tag is placed on the Feature (but not on the individual Scenarios). The CCK indicates that the tag should only be present in the Source and GherkinDocument messages.

The .NET Gherkin PickleCompiler copies those tags down to the Scenario level and thus they get emitted as tags on the Pickles that get generated. See this line of code in the PickleCompiler.cs

Does this behavior match what Reqnroll does internally during its handling of code gen from the AST? If the PickleCompiler is wrong, I'll submit a PR to remove that line. If it is correct, I'll contact the CCK team to ask for clarification and perhaps get the CCK generated samples corrected.

Resolved: Ignore this issue. This was my mistake. The existing behavior is fine.
I had added the retry(3) tag to the Feature in order to enable NUnitRetry and had forgotten to adjust the expected Message stream to include that tag in the expected output.

[clrudolphi]

build is failing while building docs. I had included two pages I had written on Messages. The TOC references them but the build can't find them? Not sure why.

I will check and fix it. Please ignore for now.

Fixed it. Mistake in file naming in the .md files I had added.