Coverage reports inacurate with multiple runs of tox

[Cynical-Optimist]

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Jan 4, 2019, 16:32

Summary

When running tox without specifying an environment list, and tox runs pytest in more than one python environment, we only collect the coverage report from the last pytest invocation.

Steps to reproduce

• Install multiple versions of python
• Run tox in your BuildStream checkout

What is the current bug behavior?

Coverage outputs a report to a .coverage file, if the .coverage file exists then it is overwritten. Effectively we only end up collecting the report from the last run.

What is the expected correct behavior?

We extract all relevant coverage.

Ideally after running tox, there is one .coverage file with combined coverage from all of the runs, this way the .coverage file feeds perfectly into the .gitlab-ci.yml mechanics which further combine that report with reports on other machines.

Or, we can change the .gitlab-ci.yml to accommodate a scenario where running tox produces multiple coverage files.

Possible fixes

Here are some fixes and notes on them:

• Use an envlist trick with additional environments

  I've spotted an example where someone setup a "testenv:start" and "testenv:end" to cleanup
  and record state between runs and then combine at the end.

  This is undesirable because we want to be able to invoke specific test environments like
  tox -e py36, tox -e py36,lint or tox -e docs

• Set the parallel option to true

  This is advertised to output a process specific report, however it seems this has no
  effect since we already set parallelism = multiprocessing which seems to automate
  the per process report collation already.

• Enable the append option

  I cant find a configuration file option for this but there is a CLI option which
  says that an existing .coverage file is not wiped but instead appended to.

  This would be dangerous if we succeed to parallelize environment runs, not sure
  if coverage expects this scenario.

[Gitlab user @BenjaminSchubert] suggests that we give each environment a separate directory to write to.

---

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 8, 2019, 17:31

Filed an enhancement request upstream with coverage.

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 8, 2019, 21:06

mentioned in commit 048c3e1294903a4588d6cc1b2b632d42dc09f901

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 9, 2019, 02:08

mentioned in commit 13a10d42fbe032cfba2e27aeecaa0153b805005e

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 9, 2019, 02:12

mentioned in commit df0f2b54e0e6cf7d8414de6d5ea316eff455a1a0

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 9, 2019, 02:48

mentioned in commit 0c72a32bbe9f29fd9d605d61cafb695918e749fe

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 9, 2019, 03:01

mentioned in commit 3e213152ba6e4a17e7314dbc36f3550a39d47a26

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 9, 2019, 15:38

mentioned in commit d7c87f801cc4f032ec7c0ba21ec1953fe277a1d4

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 9, 2019, 15:39

mentioned in commit 3a06278eebaac02711a34d3e851d374bd3e96226

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 9, 2019, 19:10

mentioned in commit 7e26cd167ae7e68559479e390d16dca99e520255

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 9, 2019, 19:54

mentioned in commit f29a099

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 9, 2019, 20:36

closed via merge request !1051

[Cynical-Optimist]

In GitLab by [Gitlab user @tristanvb] on Jan 9, 2019, 20:36

mentioned in commit 1e35243