Add global hooks, global attachments, hook types

[davidjgoss]

🤔 What's changed?

• Add new messages for test run hooks starting and finishing
• Add id to TestRunStarted so that the corresponding finish event can be tied back to it, and attachments (from test run level hooks)
• Split out TestStepResult into its own file since it's now reused

⚡️ What's your motivation?

Fixes #66 - we want to represent global hooks in the messages, because currently if they fail (thus causing the test run to fail) they aren't visible in reporting. TestRunHookStarted has a unique id like many other "started" messages, so we can represent multiple executions of the same hook in the same test run - this would happen a) in cucumber-js where global hooks run on each worker in a parallel configuration and b) in dotnet where global hooks run per-feature.

Putting an id on TestRunStarted is an approach we've discussed before and roughly agreed on in the context of global attachments per cucumber/cucumber-js#1394 (comment) - this PR doesn't address the attachments problem directly, but does lay the groundwork for it, and also means it starts to look more possible to have multiple test runs in the same stream of messages and be able to differentiate them. Note that Gherkin messages and support code aren't linked to a test run because they can be reused.

Also fixes #72 by adding a type enum to Hook messages (including types for test run hooks).

🏷️ What kind of change is this?

• 💥 Breaking change (incompatible changes to the API) because of new fields on various constructors in generated code for Java, .NET etc

📋 Checklist:

• I agree to respect and uphold the Cucumber Community Code of Conduct
• I've changed the behaviour of the code
  • I have added/updated tests to cover my changes.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly.
• Users should know about my change
  • I have added an entry to the "Unreleased" section of the CHANGELOG, linking to this pull request.

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[davidjgoss]

A bit of a chicken and egg problem with the perl tests here - they rely on the current published version of the CCK, whose messages don't include the new field, which is causing the failure. Any thoughts on how to address this @ehuelsmann?

[mpkorstanje]

A bit of a chicken and egg problem with the perl tests here - they rely on the current published version of the CCK, whose messages don't include the new field, which is causing the failure. Any thoughts on how to address this @ehuelsmann?

@ciaranmcnulty looks like the acceptance tests against the CCK are not invoked for PHP:

There were 2 skipped tests:

1) AcceptanceTest::testAllNdJsonSurvivesDecodingThenEncoding
Test for AcceptanceTest::testAllNdJsonSurvivesDecodingThenEncoding skipped by data provider
PHPUnit\Framework\SkippedTestError: <no message>

2) AcceptanceTest::testAllFileStreamsSurviveDecodingThenEncoding
Test for AcceptanceTest::testAllFileStreamsSurviveDecodingThenEncoding skipped by data provider
PHPUnit\Framework\SkippedTestError: <no message>

[mpkorstanje]

A bit of a chicken and egg problem with the perl tests here - they rely on the current published version of the CCK, whose messages don't include the new field, which is causing the failure. Any thoughts on how to address this @ehuelsmann?

@davidjgoss looking at the test I think we can change them a little so. So instead of comparing the round trip ndjson --> object --> ndjson and comparing the result we can do object --> ndjson --> object and compare that result. That way we don't need the CCK at all and the code will change along with that specific message object.

Though I think it would be even better to include a testdata folder with (for now) a single message that all languages try to round trip. That way we won't have to update N implementations manually.

[davidjgoss]

I originally added testRunStartedId to the TestCase message here. I've since removed that, having reconsidered. I think there's an argument that, given the (currently theoretical) scenario of multiple test runs in the same stream, a test case might be defined once and used in multiple test runs. So that would mean putting testRunStartedId on a TestCaseStarted message instead.

Any thoughts?

We could move forward with this change without making a decision on test cases, but it would be nice to capture as much as we can of the "test runs have an identity" thing at the same time. (Somewhat related to cucumber/compatibility-kit#112.)

[mpkorstanje]

No thoughts yet.

So that would mean putting testRunStartedId on a TestCaseStarted message instead.

I think this would a more correct solution than adding it to the TestCase. Looking at the now out of date diagram, all the execution sits on the right hand side. So it makes sense that test case execution would be enveloped by a test run execution.

https://github.com/cucumber/messages/blob/main/messages.png

[mattwynne]

I originally added testRunStartedId to the TestCase message here. I've since removed that, having reconsidered. I think there's an argument that, given the (currently theoretical) scenario of multiple test runs in the same stream, a test case might be defined once and used in multiple test runs. So that would mean putting testRunStartedId on a TestCaseStarted message instead.

Is there another existing way to know which test cases are in which test run, before it starts? I'm thinking of a system to shard a large test suite across different execution nodes, where you'd have filtered the test cases into groups for distribution. In that situation you'd want to be able to associate the compiled test case with the test run. Getting that information up-front would mean you knew how many results you were waiting for, could show a progress bar etc.

[davidjgoss]

In that situation you'd want to be able to associate the compiled test case with the test run. Getting that information up-front would mean you knew how many results you were waiting for, could show a progress bar etc.

You're right @mattwynne, and this is how the progress bar formatter in cucumber-js works, sharded or not. So yeah the link to the run does have to be at TestCase level - I'll update the PR accordingly.