nyc config `all:true` does not work #206

penx · 2020-04-21T11:59:56Z

Follow up to #194

Logs and screenshots

✗ DEBUG=code-coverage yarn coverage:integration 
yarn run v1.21.1
$ rm -rf coverage/integration && mkdir -p coverage/integration && yarn cypress run
$ /Users/penx/Development/storybook-code-coverage/node_modules/.bin/cypress run
GVA encoder info: AMD performance mode : 2 
GVA encoder info: deleteSCDMetalContext : texture cache hits: 0, misses: 0

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    4.4.0                                                                              │
  │ Browser:    Electron 80 (headless)                                                             │
  │ Specs:      1 found (spec.js)                                                                  │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  spec.js                                                                         (1 of 1)


  ✓ has a visible header (381ms)
  code-coverage parsed sent coverage +0ms
  code-coverage wrote coverage file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json +2ms
  code-coverage NYC file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json has 5 key(s) +0ms
  code-coverage 1 key /Users/penx/Development/storybook-code-coverage/src/utilities/useToggle.js file path /Users/penx/Development/storybook-code-coverage/src/utilities/useToggle.js +0ms
  code-coverage 2 key /Users/penx/Development/storybook-code-coverage/src/design-system/Toggle/index.js file path /Users/penx/Development/storybook-code-coverage/src/design-system/Toggle/index.js +0ms
  code-coverage 3 key /Users/penx/Development/storybook-code-coverage/src/design-system/GlobalStyles/index.js file path /Users/penx/Development/storybook-code-coverage/src/design-system/GlobalStyles/index.js +0ms
  code-coverage in file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json all files are not found? false +0ms
  code-coverage NYC file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json has 5 key(s) +2ms
  code-coverage combined NYC options { 'report-dir': './coverage', reporter: [ 'lcov', 'clover', 'json' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: true, all: true, include: [ 'src/**/*.js' ], exclude: [ '**/*.test.js', '**/test.js', '**/*.stories.js', '**/stories.js' ] } +0ms
  code-coverage calling NYC reporter with options { 'report-dir': '/Users/penx/Development/storybook-code-coverage/coverage', reporter: [ 'lcov', 'clover', 'json' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: true, all: true, include: [ 'src/**/*.js' ], exclude: [ '**/*.test.js', '**/test.js', '**/*.stories.js', '**/stories.js' ], tempDir: '/Users/penx/Development/storybook-code-coverage/.nyc_output' } +28ms
  code-coverage current working directory is /Users/penx/Development/storybook-code-coverage +0ms
  code-coverage after reporting, returning the report folder name /Users/penx/Development/storybook-code-coverage/coverage +56ms
  code-coverage Final coverage in /Users/penx/Development/storybook-code-coverage/coverage/coverage-final.json +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/index.js statements covered 1/1 +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/application/App.js statements covered 2/2 +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/design-system/GlobalStyles/index.js statements covered 7/7 +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/design-system/Toggle/index.js statements covered 5/5 +0ms
  code-coverage ⚠️ /Users/penx/Development/storybook-code-coverage/src/utilities/useToggle.js statements covered 6/10 +1ms

  1 passing (505ms)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     0 seconds                                                                        │
  │ Spec Ran:     spec.js                                                                          │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Video)

  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: /Users/penx/Development/storybook-code-coverage/cypress/vid    (0 seconds)
                          eos/spec.js.mp4                                                           


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  spec.js                                  497ms        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        497ms        1        1        -        -        -  

✨  Done in 8.89s.

Versions

What is this plugin's version?
3.4.1
What is Cypress version?
4.4.0
What is your operating system?
Mac OS X 10.15.4
What is the shell? zsh
What is the Node version?
12.14.1
What is the NPM version?
6.13.4
How do you instrument your application? Cypress does not instrument web application code, so you need to do it yourself.
When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? Can you paste a screenshot?

Yes

Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Can you paste at least part of it so we can see the keys and file paths?

Yes

Do you have any custom NYC settings in package.json (nyc object) or in other NYC config files

Yes, as per 'describe the bug' below

Do you run Cypress tests in a Docker container?

No

Describe the bug
If I have the following in my package.json:

  "nyc": {
    "all": true,
    "include": ["src/**/*.js"],
    "exclude": ["**/*.test.js", "**/*.stories.js"]
  }

I would expect to see files that have 0% coverage in the report, but they don't appear.

Link to the repo

https://github.com/penx/storybook-code-coverage

^ at present, you will need to update this repo to 3.4.1 and remove "report-dir": "coverage/integration", from package.json (as this setting seems to be bugged in 3.4.1)

The text was updated successfully, but these errors were encountered:

penx · 2020-04-21T13:27:36Z

I'm thinking this could be an issue with @cypress-io/instrument-cra

https://github.com/cypress-io/instrument-cra/blob/bf977b19c77799e8b9c9f4a4ee06ae42fe7678ac/index.js#L33

It looks like instrument-cra pushes the babel plugin istanbul plugin in to the webpack build, so perhaps Istanbul is only aware of files that are being built by webpack. In the example above, I have files in my project that aren't included in the webpack build, but I would want these to be included in the code coverage report (it is useful to utilise code coverage reports from integration tests to identify redundant files that are candidates for deletion).

bahmutov · 2020-04-21T13:29:00Z

Hi, I looked at the project

I have fixed the report dir, seems NYC really was using the camelCased report dir option, fixed in https://github.com/cypress-io/code-coverage/releases/tag/v3.4.2
Hmm, regarding missing files in src/utilities - I don't think NYC itself does anything differently. If after Cypress test I run

--------------------------------|---------|----------|---------|---------|-------------------
File                            | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------------------------|---------|----------|---------|---------|-------------------
All files                       |      84 |    71.43 |   66.67 |     100 |                   
 src                            |     100 |      100 |     100 |     100 |                   
  index.js                      |     100 |      100 |     100 |     100 |                   
 src/application                |     100 |      100 |     100 |     100 |                   
  App.js                        |     100 |      100 |     100 |     100 |                   
 src/design-system/GlobalStyles |     100 |       50 |     100 |     100 |                   
  index.js                      |     100 |       50 |     100 |     100 | 21                
 src/design-system/Toggle       |     100 |       75 |     100 |     100 |                   
  index.js                      |     100 |       75 |     100 |     100 | 12                
 src/utilities                  |      60 |      100 |      20 |     100 |                   
  useToggle.js                  |      60 |      100 |      20 |     100 |                   
--------------------------------|---------|----------|---------|---------|-------------------

I don't see additional files in the list. Have you seen empty files before?

for your known issue Can't configure cypress-specific nyc settings outside of package.json - I have closed this in https://github.com/cypress-io/code-coverage/releases/tag/v3.4.0

penx · 2020-04-21T13:31:55Z

I don't see additional files in the list. Have you seen empty files before?

I haven't seen Cypress report on uncovered files before, if that's what you mean? But I've only just started with this.

However, if you run yarn coverage:unit or coverage:visual-regression on the above project you'll see that Istanbul will report on uncovered files:

--------------------------------|----------|----------|----------|----------|-------------------|
File                            |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
--------------------------------|----------|----------|----------|----------|-------------------|
All files                       |    32.26 |    14.29 |    33.33 |       25 |                   |
 src                            |        0 |      100 |      100 |        0 |                   |
  index.js                      |        0 |      100 |      100 |        0 |                 6 |
 src/application                |        0 |      100 |        0 |        0 |                   |
  App.js                        |        0 |      100 |        0 |        0 |               7,8 |
 src/design-system/Alert        |        0 |      100 |        0 |        0 |                   |
  index.js                      |        0 |      100 |        0 |        0 |                 3 |
 src/design-system/GlobalStyles |        0 |        0 |        0 |        0 |                   |
  index.js                      |        0 |        0 |        0 |        0 |4,7,12,14,15,19,20 |
 src/design-system/Message      |        0 |      100 |        0 |        0 |                   |
  index.js                      |        0 |      100 |        0 |        0 |                 3 |
 src/design-system/Toggle       |        0 |        0 |        0 |        0 |                   |
  index.js                      |        0 |        0 |        0 |        0 |        4,5,6,9,10 |
 src/utilities                  |    83.33 |      100 |    83.33 |    85.71 |                   |
  addOne.js                     |        0 |      100 |        0 |        0 |                 1 |
  useToggle.js                  |      100 |      100 |      100 |      100 |                   |
--------------------------------|----------|----------|----------|----------|-------------------|

bahmutov · 2020-04-21T13:35:16Z

Yeah, I think you have identified the problem - the babel-plugin-istanbul instrumet-CRA is inserting is only instrumenting the files the webpack gives to it. Maybe cypress-io/instrument-cra#75 will change its behavior, let me see if I can upgrade that module.

PS: love your project, we definitely need to add it as an example to this repo's README

penx · 2020-04-21T13:36:58Z

PS: love your project, we definitely need to add it as an example to this repo's README

thanks! I'm working on a blog post to accompany it, but don't want to publish until I can get this thing working. I can send you a draft if you're interested

penx · 2020-04-21T15:09:48Z

I found this which is loosely related istanbuljs/babel-plugin-istanbul#105

Perhaps in task.js:resetCoverage() you can add something that adds all files to the coverage report as soon as it is initialised (with zero coverage) independently of create-react-app/webpack?

bahmutov · 2020-04-21T18:08:43Z

@penx can you try v3.5.0 - it should insert dummy placeholders for all files from package.json nyc list (if you use all:true option) into .nyc_output/out.json file before generating a report. It would look something like this:

penx · 2020-04-21T18:56:37Z

Great, that works! The uncovered files are white and don't have stats against them like number of branches and functions. One side effect of this is the lack of tests doesn't affect the overall code coverage metric. This isn't too important for me right now but perhaps something you may want to look at in the future? I could add a new feature request ticket if you like? Thanks for getting this fixed so quickly.

penx · 2020-04-21T19:02:22Z

for your known issue Can't configure cypress-specific nyc settings outside of package.json - I have closed this in https://github.com/cypress-io/code-coverage/releases/tag/v3.4.0

I was hoping to have a cypress specific file to put in cypress specific nyc settings, as other parts of the application may also use nyc.

e.g.

  "nyc": {
    "report-dir": "coverage/integration",

The report dir is a cypress specific setting.

Perhaps a nyc or code-coverage: { nyc: { section could be added to cypress.json?

bahmutov · 2020-04-21T19:06:55Z

For now, I would not want to add any more specific options to Cypress, so yes, please open a new issue about empty coverage for non-included files, ok.

penx · 2020-04-21T19:09:12Z

On a related note, I noticed when I added "reporter": ["text", "lcov"], to nyc in package.json I got the following error:

     CypressError: `cy.task('coverageReport')` failed with the following error:

> Error: ENOENT: no such file or directory, open '/Users/penx/Development/storybook-code-coverage/coverage/integration/coverage-final.json'

I'll take another look later or tomorrow and open a bug report or have a go at fixing it

bahmutov · 2020-04-21T19:27:33Z

@penx Ha, I just pushed a fix, this was just a silly bug, I forgot existsSync(...) here

const jsonReportFilename = join(folder, 'coverage-final.json')
if (!existsSync) {
  debug('Did not find final coverage file %s', jsonReportFilename)
  return
}

mukherje2 · 2022-06-07T19:40:55Z

I've got all: true in nyc config. I get the dummy entries in coverage report.

I've used this approach here https://github.com/toaditoad/code-coverage-mwe with react to instrument untested/ uncovered files. A downside of this approach is that it adds to the completion time of test runs due to webpack additionalEntries.

related issues
istanbuljs/nyc#1415
cypress-io/cypress#17230

I've a monorepo setup with cypress installed in each of the sub-package (yarn workspaces). For the sub-packages with webpack, additionalEntries param in webpack config solves this issue. But I'm stuck with a sub-package that doesn't have webpack, as I don't have additionalEntries at my disposal. Is there a way to achieve this with babel setup?

penx mentioned this issue Apr 21, 2020

nyc config all:true does not work #194

Closed

bahmutov mentioned this issue Apr 21, 2020

feat: find missing files when using all:true option #208

Merged

3 tasks

bahmutov closed this as completed Apr 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nyc config `all:true` does not work #206

nyc config `all:true` does not work #206

penx commented Apr 21, 2020 •

edited

Loading

penx commented Apr 21, 2020 •

edited

Loading

bahmutov commented Apr 21, 2020

penx commented Apr 21, 2020 •

edited

Loading

bahmutov commented Apr 21, 2020

penx commented Apr 21, 2020

penx commented Apr 21, 2020 •

edited

Loading

bahmutov commented Apr 21, 2020

penx commented Apr 21, 2020

penx commented Apr 21, 2020

bahmutov commented Apr 21, 2020

penx commented Apr 21, 2020 •

edited

Loading

bahmutov commented Apr 21, 2020

mukherje2 commented Jun 7, 2022 •

edited

Loading

nyc config all:true does not work #206

nyc config all:true does not work #206

Comments

penx commented Apr 21, 2020 • edited Loading

penx commented Apr 21, 2020 • edited Loading

bahmutov commented Apr 21, 2020

penx commented Apr 21, 2020 • edited Loading

bahmutov commented Apr 21, 2020

penx commented Apr 21, 2020

penx commented Apr 21, 2020 • edited Loading

bahmutov commented Apr 21, 2020

penx commented Apr 21, 2020

penx commented Apr 21, 2020

bahmutov commented Apr 21, 2020

penx commented Apr 21, 2020 • edited Loading

bahmutov commented Apr 21, 2020

mukherje2 commented Jun 7, 2022 • edited Loading

nyc config `all:true` does not work #206

nyc config `all:true` does not work #206

penx commented Apr 21, 2020 •

edited

Loading

penx commented Apr 21, 2020 •

edited

Loading

penx commented Apr 21, 2020 •

edited

Loading

penx commented Apr 21, 2020 •

edited

Loading

penx commented Apr 21, 2020 •

edited

Loading

mukherje2 commented Jun 7, 2022 •

edited

Loading