Rerun Test Coverage Multiple Times Produce Different Coverage Results

[ZehuaZhang]

Link to bug demonstration repository

I've followed this link in nyc npm, since we use typescript

Run tests multiple times with test coverage enabled results in different results-

Expected Behavior

Observed Behavior

Half times is 85%

Half times is 68%

Interesting fact
When the percentage is higher, which is correct, I could see each code file coverage
But when the percentage is lower, which is incorrect, when I click each code file, some files instead of showing the code and coverage for each line, shows below message

Troubleshooting steps

• still occurring when I put cache: false in my nyc config
  This does nothing for me

Environment Information

# paste the output here

System:
    OS: macOS 10.14.4
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
    Memory: 567.67 MB / 16.00 GB
  Binaries:
    Node: 12.3.1 - /usr/local/bin/node
    Yarn: 1.16.0 - /usr/local/bin/yarn
    npm: 6.9.0 - /usr/local/bin/npm
  npmPackages:
    nyc: ^14.1.1 => 14.1.1 
    source-map-support: ^0.5.12 => 0.5.12 
    ts-node: ^8.1.0 => 8.2.0 
    typescript: ^3.1.1 => 3.5.1 

[coreyfarrell]

We require a link to a minimal repository demonstrating your issue.

[tilap35173]

I created a small project that demonstrating this issue.

The link: https://github.com/tilap35173/nyc-issues

[godu]

I have the same issue with nyc/ava.

After some investigation, I identifie some troubes.
In @tilap35173 's example, we have two report whict must be merge :

{
    "./nyc-issues/src/compareNumbers.ts": {
        "path": "./nyc-issues/src/compareNumbers.ts",
        "s": {
            "0": 0,
            "1": 0,
            "2": 0,
            "3": 0,
            "4": 0,
            "5": 0,
            "6": 0
        },
		...
    },
	...
}

{
    "./nyc-issues/src/compareNumbers.ts": {
        "path": "./nyc-issues/src/compareNumbers.ts",
        "s": {
            "0": 1,
            "1": 1,
            "2": 1,
            "3": 1,
            "4": 0,
            "5": 0,
            "6": 0,
            "7": 1,
            "8": 28,
            "9": 6,
            "10": 22,
            "11": 1,
            "12": 21,
            "13": 3,
            "14": 18,
            "15": 1
        },
		...
    }
}

We can see that istanbul doesn't identify the same list of statement.
Cannot read property 'start' of undefined doesn't throws when nyc try to merge the second before the first (reports are sort by filename).

But why do we have two different mapping ? In processinfo folder, we have more informations.

{
    "uuid": "d779d080-2dfb-40e1-bb91-2097dcc97f13",
    "argv": [
        "/Users/arthurweber/.nvm/versions/node/v8.15.1/bin/node",
        "/Users/arthurweber/Sites/nyc-issues/node_modules/.bin/nyc",
        "mocha"
    ],
	...
}

{
    "uuid": "e8849878-fe5a-4639-8d58-81a8a26b69b7",
    "argv": [
        "/Users/arthurweber/.nvm/versions/node/v8.15.1/bin/node",
        "/Users/arthurweber/Sites/nyc-issues/node_modules/mocha/bin/_mocha",
        "test/**/*.spec.ts",
        "--no-config",
        "--no-package",
        "--no-opts",
        "--diff",
        "--color",
        "--extension",
        "ts",
        "--extension",
        "js",
        "--reporter",
        "min",
        "--require",
        "ts-node/register",
        "--ui",
        "bdd",
        "--slow",
        "75",
        "--timeout",
        "2000"
    ],
	...
}

In addition, I try to remove --all option. The first report isn't generated.

In my opinion, nyc try to parse source files because of --all, but it doesn't use the same process as mocha (or ava in my case).

Maybe we need to pass some --require options to nyc ?

[godu]

I may have a solution :

• .nycrc

{
	...
	"require": ["ts-node/register"]
}

[coreyfarrell]

@godu thanks for catching this. Having nyc perform the require for all transpilers is very important when you enable the nyc.all option. Without sources are not pre-proccessed by ts-node/register during processing of all: true, but ts-node/register is run by mocha which effects files when they are required by tests (so they two stages see different results). This can cause weird issues.

[godu]

We could add this configuration as default in my opinion.
istanbuljs/istanbuljs#415

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bcoe]

@coreyfarrell my hunch is this might be addressed by some of the work you've been doing around --all.

[coreyfarrell]

Yes I had expected the PR to close this issue, apparently I formatted incorrectly.

This issue should be resolved in nyc@15. Any interested parties can follow #1104 for progress on the new release.

[tananai]

@godu thank you so much! It works