Possible Generate Cucumber JSON Reports Post Test Run Execution

[bhreinb]

Hi there

I'm currently evaluating the toolset for a project I'm currently working on. I have used cucumber js with protractor before which worked very well for me in the past so hoping to do something similar with this toolset and cucumber js. I must say from starting in December the project has evolved significantly so well done on that 👍 .

I notice though that the project doesn't quiet have feature parity with cucumber js. I list key gaps from an e2e test perspective

1. support for hooks and tags (however that is almost implemented)
2. generate a cucumber JSON report after a test run (I don't think this is currently possible or I don't know how to do it)
3. define step_defintions outside of package.json (Requesting Enhancement for "Step_Definitions" location configuration in package.json to support glob patterns #70 maybe I have this wrong...see my comment in that thread)

I focus on item number 2. When I run protractor tests with cucumber js I was able to get a cucumber JSON report (typically I got this as a file). All the cucumber families (for example Cucumber-JVM & CucumberJs)
sample.json.zip

produce the same JSON.

I notice cypress supports a number of reporters formats (https://docs.cypress.io/guides/tooling/reporters.html#) from mocha, junit, etc, however I don't know how to get JSON from a cucumber js run.

The reason why I request JSON from cucumber is I can pass that JSON to this report module on a CI server (namely this module https://github.com/jenkinsci/cucumber-reports-plugin which internally uses this library https://github.com/damianszczepanik/cucumber-reporting).

Having the above but especially item number 2 opens the toolset to a wider QA audience in my opinion. Thoughts? Thanks in advance re any help with the above.

[bhreinb]

Hi folks any thoughts on the above ticket I created? Thanks in advance.

[lgandecki]

Thanks for reporting!

We need to revisit #1 soon. I'm hesitant to introduce something that will be incompatible with cypress way of doing tags, and would be very hrad to fix in a backward compatible way in a future.

#2 - I think that's mostly on cypress. Have you tried using mocha JSON reporter - https://mochajs.org/#reporters / https://docs.cypress.io/guides/tooling/reporters.html#Custom-Reporters ?

I verified that it works when I run

./node_modules/.bin/cypress run --reporter json

, it's up to cypress and cypress documentation to figure out how to get it to save to a file

#3 You can indeed define step_definitions outside package.json, we use the cosmiconfig , so you have all the options listed here: https://github.com/davidtheclark/cosmiconfig#cosmiconfig to configure the cypress-cucumber-preprocessor.

Once you figure out #2 and/or #3 I would love an PR with an update to the documentation.
At this point I'm closing this issue as I don't think there is anything here that needs to happen still to resolve it.
There is a separate issue for tags.

Feel free to reopen if you have more questions or more specific information.

Thanks again!

[bhreinb]

Hi @lgandecki

Thanks for responding.

For item 2) actually that's all on this project I'd imagine. At the end of a cucumber test run it generates a JSON report. All the cucumber runners JVM, JS, Specflow etc produce the same JSON. The JSON that cypress produces aligns to the JSON a mocha report generates which is significantly different to the JSON a cucumber runner generates hence me thinking that is item for this project.

[lgandecki]

if you want a different reporter than you need to create a mocha reporter that aligns to the cucumber format. :-)
this would be a separate project, but I understand that it might be a requirement for someone to use it.
In any case - at this point you won't get a report that would look like the one you get from other cucumber projects, because of cypress limitations (that are promised to change, but we don't know when) we have to put all scenarios in one test :(

[bhreinb]

That sounds drastic all right :(. Hmm I originally didn't think you needed to do that but looking at this a bit more now I'm not sure. Up to cucumberjs v3 you could get access to the JSON report easily enough via a cucumberjs API. Hence I was thinking this wouldn't be too bad to accomplish. That seems to be removed according to this link so I can understand why possibly a custom reporter may be needed

cucumber/cucumber-js#964

I notice this project is inspired from https://github.com/sitegeist/gherkin-testcafe which has further evolved to this project https://github.com/kiwigrid/gherkin-testcafe. I have an issue for them here

kiwigrid/gherkin-testcafe#7

be interesting to see how that gets implemented. Maybe functionality implemented there could be re-used here assuming the code bases have an element of commonality between them. Thanks for your thoughts btw.

[ralphavalon]

@bhreinb Any workaround so far? I have the same problem. I want to use a tool that parses that json to living documentation (https://github.com/rmpestano/cukedoctor), but I can't get that json anywhere. And apparently no one created some Cypress Cucumber Json Reporter.

[RobertLowe]

This would be really nice, anyone make progress on this?

[jcundill]

@RobertLowe this has been available for a while now, see https://github.com/TheBrainFamily/cypress-cucumber-preprocessor#output for details of how to configure the output of cucumber JSON files

[gretherMG]

Hello good afternoon, I am using cypress with cucumber in my local, the report cucumber-json is generated correctly, however in the azure pipeline the report is never generated, does anyone have any idea what may be happening?

@jcundill please any ideas? grateful for the help provided