Produce output JSON for integration with Cucumber for Jira

[rob-mur]

In order to integrate properly with Jira, an output file needs to be produced after the tests are ran with the following schema available here

I think I remember seeing this in the help section of the cli some versions ago but it isn't in the latest version. Can I first ask if this is a planned feature at some point?

If not, then as a description the goal is to generate a report of the tests that have been completed e.g. which steps failed and which passed, whilst conforming the output to this schema. There is a tool in that same repo which could be repurposed to check the output of this, or it could just be tested from first principles.

The repo also has some examples of invalid json outputs which could be useful in developing this feature.

I don't have much experience with the implementation of this repository, but from looking at the documentation it looks like you'd want to use the RunResult struct that's generated in Cucumber::run to generate this information, but I'd appreciate some guidance!

[rob-mur]

To flesh this out some more, here are the things we need to be able to tease out for a minimal implementation:

For each feature:

• Name
• URI (path to feature file)

For each element, in the simple case I think this is just each scenario:

• Type (Scenario/Scenario Outline)
• Line (line number in feature file)
• Name

For each step:

• Keyword (given/when/then)
• Line
• Result (not strictly required by schema but this is a bit pointless otherwise). Also requires the error message for any failure states.

Note that I've also found some example output json here

I propose the following steps:

• Locate each of the above in a way which is accessible at the end of run()
• Format the above into json
• Test this in Cucumber for JIRA by uploading some results
• If this works, loop back and add other features such as Tags/Patterns...

When I next get time I can try and jump in to these steps, but would appreciate some guidance if I'm going the wrong way/if there's something simpler!

[rob-mur]

I've started on the above by fleshing out the basic AST that would be required. Once each of the elements can be sourced and populated from cucumber as it runs, producing the output should be easy - although it will need to interface with the cli appropriately.

The AST I've come up with is as follows, suggestions welcome:

use serde::Serialize;

#[derive(Serialize)]
pub struct FeatureOutput {
    name: String,
    uri: String,
    elements: Vec<ElementOutput>,
}

#[derive(Serialize)]
pub struct ElementOutput {
    scenario_type: String,
    line: u16,
    name: String,
    steps: Vec<StepOutput>,
}

#[derive(Serialize)]
pub enum StepKeyword{
    Given,
    When,
    Then
}

#[derive(Serialize)]
pub struct StepOutput {
    keyword: StepKeyword,
    line: u16,
    result: ResultOutput
}

#[derive(Serialize)]
pub enum ResultStatus {
    Passed,
    Failed,
    Skipped,
}

#[derive(Serialize)]
pub struct ResultOutput {
    status: ResultStatus,
    error_message: Option<String>
}

Note that the final result would be a:

Vec<FeatureOutput>

[ilslv]

@rob-mur Thank you for the research!
But I've looked through cucumber-json-testdata-generator and it looks like, the spec completely skips Rule support. Do you know something about it?

UPD: Ok, I've found out that as least as of January 14, 2020 there was no support for Gherkin 6 in Jira plugin: forum answer
UPD 2: I've asked question in schema repository cucumber/cucumber-json-schema#5. Also discussed with @tyranron that we still can support this feature without official Rule support by flattening them and prepending underlying Scenarios with Rule name.

[rob-mur]

Hi, I think that's the right approach. I spoke with SmartBear a while back and they said they were targeting that schema I referenced (at least as of 2 month ago). So if we get the tool to produce output in line with that we will match what they are expecting, and then just add future functionality as required if they change it.

Hope that helps!