No coverage report generated for large project

[cklogs]

Hi there,

This library is now recommended for code coverage when using v11 of the angular-cli. We are currently using karma-coverage-istanbul-reporter because we have yet to get successful coverage reports with karma-coverage. Seeing, however, that there are deprecation warnings when using the istanbul reporter library with the v11 angular-cli, we would very much like to move off it.

Perhaps someone here can help discern what is wrong with our configuration or point us in the right direction.

I will include the karma.config, and test output debug logs for two different projects.

The first project (PROJECT-1) is a bare project that was initialized with ng new command. It was updated to Angular 11 using the ng update command from angular-cli. The point of including this project is to serve as a 'control' for the second project.

The second project (PROJECT-2) comes from a relatively large production codebase. In this codebase is roughly 5000 unit tests.

Important points

1. The angular.json test configuration block, tsconfig.spec.ts, and karma.config for both PROJECT-1 and PROJECT-2 are exactly the same. I will post the shared karma.config below.
2. v11 angular-cli still has memory management issues at build time, similar to those mentioned here. For build/run/test on PROJECT-2 we have to use the --max_old_space_size flag. Note that we are running with latest version of the angular-cli and node v12.x.

PROJECT-1 tests are run with: ng test.
PROJECT-2 tests are run with: node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng test.

3. When running tests on PROJECT-1, I get consistent coverage reports. I never see coverage reports for PROJECT-2.
4. The only discernable difference between PROJECT-1 and PROJECT-2 is the --max_old_space_size configuration property mentioned about.

Below is the karma config file shared by the two projects and the debug logs that are output when running tests.

Karma config

// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html

module.exports = function (config) {
  config.set({
    basePath: '',
    frameworks: ['jasmine', '@angular-devkit/build-angular'],
    plugins: [
      require('karma-spec-reporter'),
      require('karma-jasmine'),
      require('karma-chrome-launcher'),
      require('karma-coverage'),
      require('@angular-devkit/build-angular/plugins/karma')
    ],
    client: {
      clearContext: false // leave Jasmine Spec Runner output visible in browser
    },
    coverage: {
      dir: require('path').join(__dirname, '../coverage/karma-coverage'),
      reports: ['html', 'lcovonly', 'text-summary']
    },
    reporters: ['dots', 'spec', 'coverage'],
    specReporter: {
	maxLogLines: 5, // limit number of lines logged per test
	suppressErrorSummary: true, // do not print error summary
	suppressFailed: false, // do not print information about failed tests
	suppressPassed: true, // do not print information about passed tests
	suppressSkipped: true, // do not print information about skipped tests
	showSpecTiming: false, // print the time elapsed for each spec
	failFast: false // test would finish with error when a first fail occurs.
     },
    customLaunchers: {
	'CustomChromeHeadless': {
		base: 'ChromeHeadless',
		flags: [
			'--disable-gpu',
			'--disable-web-security',
			'--no-sandbox',
			'--remote-debugging-port=9222'
		],
		debug: true
	}
    },
	browsers: ['CustomChromeHeadless'],
	browserDisconnectTolerance: 2,
	browserNoActivityTimeout: 300000, // five minutes
	browserDisconnectTimeout: 60000,
	captureTimeout: 300000,
	hostname: '127.0.0.1',
	port: 9876,
	colors: true,
	logLevel: config.LOG_DEBUG,
	autoWatch: false,
	singleRun: true,
	restartOnFileChange: false
  });
};

PROJECT-1-debug-output.txt
PROJECT-2-debug-output.txt

In the PROJECT-2 output you will see only ~20 tests ran. This is because I used an fdescribe to run a single test file so as to reduce the noise a little. The same result--no coverage--happens with or without this change.

The point of failure seems to be here:

Thanks in advance for your support.

[alan-agius4]

@cklogs, karma-coverage needs to be configured using the coverageReporter property, which is missing from your configuration.

Can you please try to change the configuration like the below;

// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html

module.exports = function (config) {
    config.set({
        basePath: '',
        frameworks: ['jasmine', '@angular-devkit/build-angular'],
        plugins: [
            require('karma-spec-reporter'),
            require('karma-jasmine'),
            require('karma-chrome-launcher'),
            require('karma-coverage'),
            require('@angular-devkit/build-angular/plugins/karma')
        ],
        client: {
            clearContext: false // leave Jasmine Spec Runner output visible in browser
        },
+       coverageReporter: {
+           dir: require('path').join(__dirname, './coverage/karma-coverage'),
+           subdir: '.',
+           reporters: [
+               { type: 'html' },
+               { type: 'text-summary' }
+            ]
+        },
-       coverage: {
-           dir: require('path').join(__dirname, '../coverage/karma-coverage'),
-           reports: ['html', 'lcovonly', 'text-summary']
-       },
        reporters: ['dots', 'spec', 'coverage'],
        specReporter: {
            maxLogLines: 5, // limit number of lines logged per test
            suppressErrorSummary: true, // do not print error summary
            suppressFailed: false, // do not print information about failed tests
            suppressPassed: true, // do not print information about passed tests
            suppressSkipped: true, // do not print information about skipped tests
            showSpecTiming: false, // print the time elapsed for each spec
            failFast: false // test would finish with error when a first fail occurs.
        },
        customLaunchers: {
            'CustomChromeHeadless': {
                base: 'ChromeHeadless',
                flags: [
                    '--disable-gpu',
                    '--disable-web-security',
                    '--no-sandbox',
                    '--remote-debugging-port=9222'
                ],
                debug: true
            }
        },
        browsers: ['CustomChromeHeadless'],
        browserDisconnectTolerance: 2,
        browserNoActivityTimeout: 300000, // five minutes
        browserDisconnectTimeout: 60000,
        captureTimeout: 300000,
        hostname: '127.0.0.1',
        port: 9876,
        colors: true,
        logLevel: config.LOG_DEBUG,
        autoWatch: false,
        singleRun: true,
        restartOnFileChange: false
    });
};

More information about karma-coverage configuration can be found here: https://github.com/karma-runner/karma-coverage/blob/master/docs/configuration.md

[cklogs]

@alan-agius4

I tried exactly as is suggested above and coverage reports are still not being generated.
Perhaps there is something more I'm missing?

[alan-agius4]

Without a reproduction it's hard to tell what's happening. You can try to debug

karma-coverage/lib/reporter.js

Line 224 in 1d34a56

this.executeReport = async function (reporterConfig, browser) {

to see what's happening.

[cklogs]

I'm still trying to demystify what is going on but I have observed the following:
For these particular methods:

karma-coverage/lib/reporter.js

Lines 185 to 201 in 1d34a56

	this.onRunStart = function (browsers) {
	coverageMaps = Object.create(null)
	// TODO(vojta): remove once we don't care about Karma 0.10
	if (browsers) {
	browsers.forEach(this.onBrowserStart.bind(this))
	}
	}
	this.onBrowserStart = function (browser) {
	var startingMap = {}
	if (includeAllSources) {
	startingMap = globalCoverageMap.get()
	}
	coverageMaps[browser.id] = istanbulLibCoverage.createCoverageMap(startingMap)
	}

The browsers object on line 185 is defined as:

BrowserCollection {
  browsers: [],
  emitter: EventEmitter {
    _events: [Object: null prototype] {},
    _eventsCount: 0,
    _maxListeners: undefined,
    [Symbol(kCapture)]: false
  }
}

This means that if (browsers) { executes.

However, this.onBrowserStart is never called during the entire execution of the tests.

Once the tests have all ran, this function is called:

karma-coverage/lib/reporter.js

Lines 203 to 210 in 1d34a56

	this.onBrowserComplete = function (browser, result) {
	var coverageMap = coverageMaps[browser.id]
	if (!coverageMap) return
	if (!result || !result.coverage) return
	coverageMap.merge(result.coverage)
	}

The parameter browser is defined as:

Browser {
  id: '35804940',
  fullName: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/89.0.4389.90 Safari/537.36',
  name: 'Chrome Headless 89.0.4389.90 (Windows 10)',
  lastResult: BrowserResult {
    startTime: 1617310253880,
    total: 0,
    success: 6032,
    failed: 0,
    skipped: 0,
    totalTime: 628830,
    netTime: 437749,
    error: false,
    disconnected: false
  },
  disconnectsCount: 0,
  activeSockets: [
    Socket {
      _events: [Object: null prototype],
      _eventsCount: 8,
      _maxListeners: undefined,
      nsp: [Namespace],
      client: [Client],
      acks: Map {},
      fns: [],
      flags: {},
      _rooms: Set {},
      server: [Server],
      adapter: [Adapter],
      id: 'eEVk_hCsxP6b3WzbAAAD',
      connected: true,
      disconnected: false,
      handshake: [Object],
      [Symbol(kCapture)]: false
    }
  ],
  noActivityTimeout: 300000,
  singleRun: true,
  clientConfig: {
    args: [],
    useIframe: true,
    runInParent: false,
    captureConsole: true,
    clearContext: true,
    originalArgs: []
  },
  collection: BrowserCollection {
    browsers: [ [Circular] ],
    emitter: Server {
      _events: [Object: null prototype],
      _eventsCount: 17,
      _maxListeners: undefined,
      log: [Logger],
      loadErrors: [],
      _injector: [Injector],
      _boundServer: [Server],
      _fileList: [FileList],
      [Symbol(kCapture)]: false
    }
  },
  emitter: Server {
    _events: [Object: null prototype] {
      load_error: [Function],
      exit: [Array],
      run_complete: [Array],
      browser_complete: [Array],
      file_list_modified: [Array],
      run_start: [Array],
      browser_start: [Array],
      browser_error: [Array],
      browser_log: [Array],
      spec_complete: [Array],
      spec_failure: [Function],
      browsers_change: [Function],
      browser_register: [Function],
      stop: [Function],
      browser_restart_failure: [Function],
      browser_complete_with_no_more_retries: [Function],
      browser_process_failure: [Function]
    },
    _eventsCount: 17,
    _maxListeners: undefined,
    log: Logger {
      category: 'karma-server',
      context: {},
      parseCallStack: [Function: defaultParseCallStack]
    },
    loadErrors: [],
    _injector: Injector {
      _providers: [Object: null prototype],
      _instances: [Object: null prototype],
      get: [Function: get],
      invoke: [Function: invoke],
      instantiate: [Function: instantiate],
      createChild: [Function: createChild]
    },
    _boundServer: Server {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      _connections: 0,
      _handle: [TCP],
      _usingWorkers: false,
      _workers: [],
      _unref: false,
      allowHalfOpen: false,
      pauseOnConnect: false,
      _connectionKey: '4:0.0.0.0:9876',
      [Symbol(kCapture)]: false,
      [Symbol(asyncId)]: 13
    },
    _fileList: FileList {
      _patterns: [Array],
      _excludes: [Array],
      _emitter: [Circular],
      _preprocess: [AsyncFunction: preprocess],
      buckets: [Map],
      _refreshing: [Promise],
      _emitModified: [Function]
    },
    [Symbol(kCapture)]: false
  },
  socket: Socket {
    _events: [Object: null prototype] {
      start: [Function],
      info: [Function],
      karma_error: [Function],
      result: [Function],
      complete: [Function],
      error: [Function],
      register: [Function],
      disconnect: [Function]
    },
    _eventsCount: 8,
    _maxListeners: undefined,
    nsp: Namespace {
      _events: [Object: null prototype],
      _eventsCount: 1,
      _maxListeners: undefined,
      sockets: [Map],
      _fns: [],
      _rooms: Set {},
      _flags: {},
      _ids: 0,
      server: [Server],
      name: '/',
      adapter: [Adapter],
      [Symbol(kCapture)]: false
    },
    client: Client {
      sockets: Map {},
      nsps: Map {},
      server: [Server],
      conn: [Socket],
      encoder: Encoder {},
      decoder: [Decoder],
      id: '2DlQ3r3gH89vkFHCAAAA',
      onclose: [Function: bound onclose],
      ondata: [Function: bound ondata],
      onerror: [Function: bound onerror],
      ondecoded: [Function: bound ondecoded],
      connectTimeout: undefined
    },
    acks: Map {},
    fns: [],
    flags: {},
    _rooms: Set {},
    server: Server {
      _events: [Object: null prototype] {},
      _eventsCount: 0,
      _maxListeners: undefined,
      _nsps: [Map],
      parentNsps: Map {},
      _path: '/socket.io',
      clientPathRegex: /^\/socket\.io\/socket\.io(\.min|\.msgpack\.min)?\.js(\.map)?$/,
      _connectTimeout: 45000,
      _serveClient: true,
      _parser: [Object],
      encoder: Encoder {},
      _adapter: [class Adapter extends EventEmitter],
      sockets: [Namespace],
      opts: [Object],
      eio: [Server],
      httpServer: [Server],
      engine: [Server],
      [Symbol(kCapture)]: false
    },
    adapter: Adapter {
      _events: [Object: null prototype] {},
      _eventsCount: 0,
      _maxListeners: undefined,
      nsp: [Namespace],
      rooms: [Map],
      sids: [Map],
      encoder: Encoder {},
      [Symbol(kCapture)]: false
    },
    id: 'tpLs3SFHdGeiHnz_AAAB',
    connected: false,
    disconnected: true,
    handshake: {
      headers: [Object],
      time: 'Thu Apr 01 2021 13:50:53 GMT-0700 (Pacific Daylight Time)',
      address: '127.0.0.1',
      xdomain: false,
      secure: false,
      issued: 1617310253837,
      url: '/socket.io/?EIO=4&transport=polling&t=NYFD0BQ',
      query: [Object: null prototype],
      auth: {}
    },
    [Symbol(kCapture)]: false
  },
  timer: {
    setTimeout: [Function: setTimeout],
    clearTimeout: [Function: clearTimeout]
  },
  disconnectDelay: 60000,
  log: Logger {
    category: 'Chrome Headless 89.0.4389.90 (Windows 10)',
    context: {},
    parseCallStack: [Function: defaultParseCallStack]
  },
  noActivityTimeoutId: Timeout {
    _idleTimeout: 300000,
    _idlePrev: [TimersList],
    _idleNext: [TimersList],
    _idleStart: 983207,
    _onTimeout: [Function],
    _timerArgs: undefined,
    _repeat: null,
    _destroyed: false,
    [Symbol(refed)]: true,
    [Symbol(asyncId)]: 1889860,
    [Symbol(triggerId)]: 1889856
  },
  pendingDisconnect: Timeout {
    _idleTimeout: -1,
    _idlePrev: null,
    _idleNext: null,
    _idleStart: 385875,
    _onTimeout: null,
    _timerArgs: undefined,
    _repeat: null,
    _destroyed: true,
    [Symbol(refed)]: true,
    [Symbol(asyncId)]: 1809518,
    [Symbol(triggerId)]: 1809514
  },
  state: 'CONNECTED'
}

coverageMaps are defined as:

[Object: null prototype] {}

The result is a large object containing coverage results for each file. One such example:

   'C:\\app\\shared\\helpers\\url-serializer.ts': {
      path: 'C:\\app\\shared\\helpers\\url-serializer.ts',
      statementMap: [Object],
      fnMap: [Object],
      branchMap: [Object],
      s: [Object],
      f: [Object],
      b: [Object],
      inputSourceMap: [Object],
      _coverageSchema: '1a1c01bbd47fc00a2c39e90264f33305004495a9',
      hash: 'b12799a1103f348e9c0f91bb2cb5bbf81c79d909'
    }

Is there anything more you would like to see?

[cklogs]

@alan-agius4 When you have a chance, could you please review my updates?
Advance thanks!

[alan-agius4]

I'll leave the triaging to someone from the karma-coverage team.

[cklogs]

To the maintainers of this repo:

I have a small suspicion that,

1. the sheer size of our project and,
2. the associated long compile time while running tests

have something to do with coverage failing to be generated. But this is only a hunch.

I cannot share or anonymize our code base, but if there is a large open repository that you would suggest I try and reproduce this issue from, I will happily try.

Aside from that, if you have any insights on the above issue, please let me know here or contact me directly.

Thanks again.

[WebMex]

We also have a big project with 3000+ unit tests. And I can see coverage reports generated only on powerful machines. On our ci server and on slow machines it's not generated.

[cklogs]

@WebMex
Thank you for the additional data point!

Do you plan to continue using karma-coverage, or have you sought out a different library? We have not been able to move onto karma-coverage (from karma-coverage-instanbul-reporter) because of the above issue, but we are open to other coverage generators.

[WebMex]

@cklogs I have rolled back to karma-coverage-instanbul-reporter so far and waiting for the fix. Despite the warning message from angular it is still possible to use it.

[cklogs]

@alan-agius4

So far I have had little luck getting feedback from any of the maintainers of this library.
Given karma-coverage is now the defacto standard coverage library for Angular, it may be necessary to raise this up to the greater Angular team.

Please advise on how to most effectively do this.

Thanks in advance for your help!

[cklogs]

@WebMex

Thanks to your tip, I tested karma-coverage on a faster box (Ryzen 7 PRO / 32 GB RAM), and I am getting consistent coverage results there. Still need this fix for our deployment servers to generate coverage, but I'm hoping these multiple data points give the maintainers a lead on the issue.

[alan-agius4]

This would be a hard one to track down without a reproduction. I tried it on a seed project and generated a large number of tests but I didn't manage to replicate.

Unfortunately, I am not too familiar with this plugin to get to the bottom of this without being able to reproduce the issue.