State of skipped scenarios in JSON report

[csvtuda]

Hi,

first of all, thanks for the great plugin. I maintain a plugin for uploading test run results to a test management tool and have recently been asked why tests marked as skipped (@skip) do not show up in the uploaded results (see Qytera-Gmbh/cypress-xray-plugin#341). My plugin relies heavily on the JSON report that your your plugin outputs and uploads more or less as is.

A MWE revealed that skipped tests do not show up at all in the produced report:

# cucumber.feature
Feature: Testing skips

    @skip
    Scenario: Scenario 1
        Given a step

    Scenario: Scenario 2
        Given a step

// cucumber-report.json
[
  {
    "description": "",
    "elements": [
      {
        "description": "",
        "id": "testing-skips;scenario-2",
        "keyword": "Scenario",
        "line": 4,
        "name": "Scenario 2",
        "steps": [
          {
            "arguments": [],
            "keyword": "Given ",
            "line": 5,
            "name": "a step",
            "match": {
              "location": "not available:0"
            },
            "result": {
              "status": "passed",
              "duration": 26000000
            }
          }
        ],
        "tags": [],
        "type": "scenario"
      }
    ],
    "id": "testing-skips",
    "line": 1,
    "keyword": "Feature",
    "name": "Testing skips",
    "tags": [],
    "uri": "cucumber.feature"
  }
]

---

I then searched this repository for any issues discussing this topic. If I understand correctly, the timeline was as follows:

1. skipped scenarios were not initially included
2. #329 was merged, which then added skipped tests back into the reports and introduced undesired side effects
3. mitigation attempt: feat(createtestfromscenario.js): add feature to hide pending scenarios from test run #474
4. mitigation attempt: Make logging skipped scenarios optional #485

There's talks about reverting #329 in the mitigation attempts, but I haven't found any more information on that. I assume the changes have been reverted at some point?

---

Question: What's the final decision on how and if skipped tests should be included in the JSON report? I'm not asking for a feature, just for clarification so I can handle my own issue accordingly.

[badeball]

What's the final decision on how and if skipped tests should be included in the JSON report?

Skipped tests should ideally be included in the JSON report and marked as such. However, this is technically unfeasible without also introducing the mentioned undesired side effects.

Lack of status of skipped tests also affects HTML reports, which is reported here.

This would ideally be solved with more fine-grained plugin events, which has been previously dicussed, but never made it upstream. There is a proposal, but as explained in #1041 (comment), I don't think anything will ever happen with this, simply because Cypress hasn't made any meaningful, positive changes towards plugin developers in a long time.

[csvtuda]

Thanks for the quick reply and the further related links. It's unfortunate, but I can obviously instead leverage Cypress's own run results, which include the test in its (extra confusing...) pending state:

"tests": [
  {
    "attempts": [
      {
        "state": "pending"
      }
    ],
    "displayError": null,
    "duration": 39,
    "state": "pending",
    "title": [
      "Testing skips",
      "Scenario 1"
    ]
  }
],

So I guess I'll be joining the waitlist of your proposal. Fingers crossed for some updates in that regard.

[badeball]

I couldn't quite shake this away, so I decided to take another stab at it and made it work despite the lack of fine-grained plugin events. Skipped scenarios (however skipped) are now included in the reports (hopefully). I've released this as v20.1.0.

[csvtuda]

After a little bit of experimentation, the new version seems to reliably include the skipped scenarios as well, thanks a lot!