Coverage.py 7.0.2 reports fewer statements than 7.0.1 under Python 3.7

[bhrutledge]

Describe the bug

Twine's scheduled CI failed this morning, because the coverage percentage dropped below the threshold under Python 3.7 (GHA), with 1785 statements executed. The day before, it passed (GHA), with 1807 statements executed. There were no commits in between. The only difference that I'm aware of is the release of Coverage.py 7.0.2, but I don't see any red flags in the release notes.

To Reproduce

1. What version of Python are you using?

   3.7 and 3.8

2. What version of coverage.py shows the problem? The output of coverage debug sys is helpful.

   7.0.2

3. What versions of what packages do you have installed? The output of pip freeze is helpful.

   % .tox/py37/bin/pip freeze

    attrs==22.2.0
    bleach==5.0.1
    build==0.9.0
    certifi==2022.12.7
    charset-normalizer==2.1.1
    commonmark==0.9.1
    coverage==7.0.2
    docutils==0.19
    exceptiongroup==1.1.0
    idna==3.4
    importlib-metadata==6.0.0
    importlib-resources==5.10.2
    iniconfig==1.1.1
    jaraco.classes==3.2.3
    keyring==23.13.1
    more-itertools==9.0.0
    packaging==22.0
    pep517==0.13.0
    pkginfo==1.9.2
    pluggy==1.0.0
    pretend==1.0.9
    Pygments==2.14.0
    pytest==7.2.0
    pytest-socket==0.5.1
    readme-renderer==37.3
    requests==2.28.1
    requests-toolbelt==0.10.1
    rfc3986==2.0.0
    rich==13.0.0
    six==1.16.0
    tomli==2.0.1
    twine @ file:///Users/bhrutledge/Dev/twine/.tox/.tmp/package/35/twine-4.0.3.dev21%2Bg89fba32.d20230103.tar.gz
    typing_extensions==4.4.0
    urllib3==1.26.13
    webencodings==0.5.1
    zipp==3.11.0
   

4. What code shows the problem?

   https://github.com/pypa/twine/tree/89fba32c3cf56ff6b02c691154b3de7faf7def04

5. What commands did you run?

   I'm able to reproduce this locally by running tox -e py37 and changing the coverage requirement in deps (e.g. to coverage==7.0.1 and coverage<7).

Expected behavior

The number of executed statements should be the same as previous versions.

Additional context

There's no change in statements from Coverage.py 6.5.0 to 7.0.1; both report 1807.

Also maybe worth noting that py37 has always seemed to be lower; py38 and greater seem to consistently execute 1856 statements, with no change across Coverage.py versions.

[nedbat]

Could you produce an HTML report for each version of coverage? Then we can see where the differences are. I don't know why the number of statements would differ.

[bhrutledge]

Here you go, with the statements per version:

Version	Total	twine/	tests/
twine-htmlcov-701-py37.zip	1807	898	909
twine-htmlcov-703-py37.zip	1785	877	908
twine-htmlcov-703-py38.zip	1856	898	958

It's interesting how the twine/ statements are the same between 701-py37 and 703-py38, but the tests/ are different.

I omitted 701-py38 because it has the results as 703-py38.

[nedbat]

I have spent some time bisecting, etc, and have narrowed this down to an "impossible" situation.

With coverage.py 7.0.1, twine/settings.py in the HTML report is reported as having 81 statements.

If I add from __future__ import annotations to my parser.py, then it's reported as having 75 statements. How can this be? parser.py has no type annotations, so how can a __future__ import about annotations change anything?

@bhrutledge can you confirm my results? Use this to install it:

pip install git+https://github.com/nedbat/coveragepy@nedbat/wtf2-1524#egg=coverage==0.0

Here's the diff: https://github.com/nedbat/coveragepy/compare/7.0.1..nedbat/wtf2-1524

[sco1]

Perhaps it has something to do with the decorator parsing? Looking at PythonParser.statemtents for Twine's settings.py with & without the future import in parser.py gives a difference of [129, 134, 145, 149, 156, 274], which are all line numbers around the decorators.

edit: Still doing digging but it looks like it starts to diverge when updating raw_statements here:

coveragepy/coverage/parser.py

Lines 195 to 197 in c5cda3a

	if not empty:
	byte_parser = ByteParser(self.text, filename=self.filename)
	self.raw_statements.update(byte_parser._find_statements())

Without the future import there are 189 raw statements, dropping to 178 once the future import is added.

[nedbat]

The mystery was solved by @JelleZijlstra in the Python Discord: the compile builtin inherits flags from the code calling it, so the twine code had the __future__ applied. The twine code has annotations, but does not use the __future__ import, so it was changed as a result.

I will add the flag to compile() that disables the inheritance, and then dig into how the annotations change the coverage reporting as a separate issue.

[nedbat]

This is fixed in commit 880a64a.

[nedbat]

This is now released as part of coverage 7.0.5.

[bhrutledge]

Thank you! Confirmed this fixes the issue.