test results skipped when using mocha's retries api

[zhouhua]

Hi Adam,

I have a problem when using mocha retries api.

Here's a spec file:

const expect = require('chai').expect;
let time = 0;
describe('Test cases', function () {
    it('retry', function () {
        this.retries(3);
        time++;
        expect(time).to.be.equal(4);
    });

    it('retry', function () {
        time++;
        expect(time).to.be.equal(8);
    }).retries(3);

    it('something else', () => {
    });
});

And my test script is:

mocha -R mochawesome test.spec.js

The result in console is correct:

 Test cases
    ✓ retry
    ✓ retry
    ✓ something else


  3 passing (11ms)

However, the generated html report shows a quite different result:

Why these two test cases using retries be skipped?

[adamgruber]

This is due to the way mocha handles retries. When mocha retries a test, it clones the test and adds it to the array of tests to be run. It doesn't mark the retried test in any special way or emit an event for retries so there is nothing for the reporter to hook into to know that the test was retried. Further, Mochawesome generates its report based on the suite object emitted in the suite end event which does not contain these cloned tests.

There is some more discussion around this here: mochajs/mocha#2188

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Stephane99]

Hi,
The problem is still present in mochawesome.
Retried tests finished in failed state don't appear in the report but are counted in the total number of failed tests.

I tried anothers reports like "mocha-simple-html-reporter" and this last one is able to display retried test in error.

Here is the test:

describe( 'Test retries', function () {
this.retries(1);
it('Test 1 => OK', function () {
expect(1).to.equal(1);
});
it('Test 2 => KO', function () {
expect(1).to.equal(0);
});
});

And here is the resulting report from 'mocha-simple-html-reporter'.
https://gist.github.com/Stephane99/64ac7a97d36942241f6bf0ee4fce6a8e

I hope this post can help to perform your very awesome mocha report 👍

[adamgruber]

Looked into this again but I'm still not sure there's an easy solve. Problem is that retries are kind of a hack. They are clones of the original tests. They don't relate back to the original in any way and are not added to the parent suite's list of tests. When a retry test passes or fails, the stats object is modified, however the original test is not updated to reflect the status and does not count toward the stats.

[juergba]

The Mocha runner emits an EVENT_TEST_RETRY event, in case of cloning and retrying a test.

[dcr007]

@adamgruber -
Wanted to check if we have a workaround to get this work, any suggestions appreciated ? :

while using this.retries(), I see the json results showing null for the elements state and speed the elements pass , fail as false and err is empty, also as the result of which the html report looks incorrect . Here's my test

describe(`Page Load checks `, async function ()  {

it('Page Check', async function () {
        this.retries(1); 
        presentationsPage = await navMenu.loadPresentationsPage();
        const result = await presentationsPage.isPresentationsPageLoadComplete();
        expect(result).to.equal(true, 'Expected : Presentation Page');
    }) ;
 } );

and the json results

{
              "title": "Page Check",
              "fullTitle": "Page Loads  Check",
              "timedOut": false,
              "duration": 32014,
              "state": null,
              "speed": null,
              "pass": false,
              "fail": false,
              "pending": false,
              "context": null,
              "code": "__awaiter(this, void 0, void 0, function...",
              "err": {},
              "uuid": "6971b4af-b327bda4ffa8",
              "parentUUID": "3345841e-6a5f-19fee98",
              "isHook": false,
              "skipped": true
            }

[adamgruber]

This no longer appears to be an issue starting with mocha 7.1.0