Started to raise "AssertionError: Oops, byte_incr = 2, line_incr = 2" with switch to coverage 7.6.5 py 3.9

[yarikoptic]

Describe the bug

We finally decided to ditch python 3.8 and thus switched CI runs to 3.9 in

• Drop Python 3.8 datalad/datalad#7682

but many runs failed with similar problem(s), e.g. from https://github.com/datalad/datalad/actions/runs/11844567886/job/33008095384?pr=7682

2024-11-14T20:05:30.1119963Z �[36;1mpython -m coverage xml�[0m
2024-11-14T20:05:30.1177922Z shell: /usr/bin/bash -e {0}
2024-11-14T20:05:30.1178206Z env:
2024-11-14T20:05:30.1178509Z   pythonLocation: /opt/hostedtoolcache/Python/3.9.20/x64
2024-11-14T20:05:30.1179046Z   PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.20/x64/lib/pkgconfig
2024-11-14T20:05:30.1179571Z   Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.20/x64
2024-11-14T20:05:30.1180054Z   Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.20/x64
2024-11-14T20:05:30.1180527Z   Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.20/x64
2024-11-14T20:05:30.1181008Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.20/x64/lib
2024-11-14T20:05:30.1181401Z   DL_PIP_INSTALLS: mock
2024-11-14T20:05:30.1181667Z   DL_APT_INSTALLS: dcm2niix
2024-11-14T20:05:30.1181972Z   DL_REPO: datalad/datalad-container
2024-11-14T20:05:30.1182325Z   DL_TESTER: pytest
2024-11-14T20:05:30.1182569Z   DL_NEED_SINGULARITY: 1
2024-11-14T20:05:30.1182840Z   DL_PACKAGE: datalad_container
2024-11-14T20:05:30.1183131Z ##[endgroup]
2024-11-14T20:05:32.0063609Z Traceback (most recent call last):
2024-11-14T20:05:32.0065160Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2024-11-14T20:05:32.0066400Z     return _run_code(code, main_globals, None,
2024-11-14T20:05:32.0067530Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/runpy.py", line 87, in _run_code
2024-11-14T20:05:32.0068599Z     exec(code, run_globals)
2024-11-14T20:05:32.0070188Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/__main__.py", line 10, in <module>
2024-11-14T20:05:32.0071416Z     sys.exit(main())
2024-11-14T20:05:32.0075327Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/cmdline.py", line 970, in main
2024-11-14T20:05:32.0076573Z     status = CoverageScript().command_line(argv)
2024-11-14T20:05:32.0078003Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/cmdline.py", line 730, in command_line
2024-11-14T20:05:32.0079144Z     total = self.coverage.xml_report(
2024-11-14T20:05:32.0080387Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/control.py", line 1210, in xml_report
2024-11-14T20:05:32.0081912Z     return render_report(self.config.xml_output, XmlReporter(self), morfs, self._message)
2024-11-14T20:05:32.0083963Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/report_core.py", line 62, in render_report
2024-11-14T20:05:32.0085247Z     ret = reporter.report(morfs, outfile=outfile)
2024-11-14T20:05:32.0086608Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/xmlreport.py", line 106, in report
2024-11-14T20:05:32.0087913Z     for fr, analysis in get_analysis_to_report(self.coverage, morfs):
2024-11-14T20:05:32.0089595Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/report_core.py", line 101, in get_analysis_to_report
2024-11-14T20:05:32.0090928Z     analysis = coverage._analyze(morf)
2024-11-14T20:05:32.0092220Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/control.py", line 946, in _analyze
2024-11-14T20:05:32.0093805Z     return analysis_from_file_reporter(data, self.config.precision, file_reporter, filename)
2024-11-14T20:05:32.0095699Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/results.py", line 31, in analysis_from_file_reporter
2024-11-14T20:05:32.0096977Z     statements = file_reporter.lines()
2024-11-14T20:05:32.0098246Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/python.py", line 195, in lines
2024-11-14T20:05:32.0099401Z     return self.parser.statements
2024-11-14T20:05:32.0101464Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/python.py", line 190, in parser
2024-11-14T20:05:32.0102689Z     self._parser.parse_source()
2024-11-14T20:05:32.0104027Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/parser.py", line 267, in parse_source
2024-11-14T20:05:32.0105164Z     self._raw_parse()
2024-11-14T20:05:32.0106337Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/parser.py", line 199, in _raw_parse
2024-11-14T20:05:32.0107634Z     self.raw_statements.update(byte_parser._find_statements())
2024-11-14T20:05:32.0109207Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/parser.py", line 459, in _find_statements
2024-11-14T20:05:32.0110469Z     yield from bp._line_numbers()
2024-11-14T20:05:32.0111801Z   File "/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/parser.py", line 440, in _line_numbers
2024-11-14T20:05:32.0113208Z     assert line_num != last_line_num, f"Oops, {byte_incr = }, {line_incr = }"
2024-11-14T20:05:32.0114062Z AssertionError: Oops, byte_incr = 2, line_incr = 2
2024-11-14T20:05:32.1348945Z ##[error]Process completed with exit code 1.

in the logs before running that command, while running actual tests

2024-11-14T20:05:25.0073912Z bb -> .datalad/environments/bb/image
2024-11-14T20:05:27.0085054Z /opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/report_core.py:116: CoverageWarning: Couldn't parse '/home/runner/work/datalad/datalad/datalad/distribution/tests/test_siblings.py': Oops, byte_incr = 2, line_incr = 2 (couldnt-parse)
2024-11-14T20:05:27.0087672Z   coverage._warn(msg, slug="couldnt-parse")
2024-11-14T20:05:27.8205359Z /opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/coverage/report_core.py:116: CoverageWarning: Couldn't parse '/home/runner/work/datalad/datalad/datalad/local/subdatasets.py': Oops, byte_incr = 2, line_incr = 2 (couldnt-parse)
2024-11-14T20:05:27.8207952Z   coverage._warn(msg, slug="couldnt-parse")
2024-11-14T20:05:29.9715809Z PASSED

so most likely coverage has issues parsing these two files

• https://github.com/datalad/datalad/blob/maint/datalad/distribution/tests/test_siblings.py
• https://github.com/datalad/datalad/blob/maint/datalad/local/subdatasets.py

but unclear with what and why not before.

To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

1. What version of Python are you using?

3.9

1. What version of coverage.py shows the problem? The output of coverage debug sys is helpful.

2024-11-14T20:02:02.1498025Z Collecting coverage (from datalad==1.1.3+35.gab7380b9e)
2024-11-14T20:02:02.1649949Z   Downloading coverage-7.6.5-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.2 kB)

before this we had 3.8 and it was installing e.g coverage-7.6.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2 and we had no such issues

1. What versions of what packages do you have installed? The output of pip freeze is helpful.

here is some

dependencies

• annexremote: 1.6.6
• boto3: 1.35.60
• humanize: 4.11.0
• iso8601: 2.1.0
• keyring: 25.5.0
• keyrings.alt: 5.0.2
• msgpack: 1.1.0
• platformdirs: 4.3.6
• requests: 2.32.3

1. What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix.
2. What commands should we run to reproduce the problem? Be specific. Include everything, even git clone, pip install, and so on. Explain like we're five!

failed to replicate locally.

Expected behavior

Not to raise AssertionError ;)

More information on problem with parsing -- at least point to the line numbers or even visualize them.

Additional context
Add any other context about the problem here.

[yarikoptic]

FWIW, pinning coverage to 7.6.1 provided a workaround, so should be related to diff between 7.6.1 and 7.6.5.

[nedbat]

Yes, the changelog for 7.6.5 mentions:

• refactor: some code unreachable code paths in parser.py were changed to asserts. If you encounter any of these, please let me know!

I'll take a look at the files you've indicated to understand why they are different than most.

[nedbat]

This is fixed in commit 98939c9 and released as part of coverage 7.6.6.

[nedbat]

If anyone finds another similar assert in 7.6.6, please provide details.