With relative_files = true, I'm still getting absolute paths in .coverage due to $PWD-sensitivity

[jab]

(Same symptom as #1147 but different cause.)

I need to produce relative paths in the coverage data that coverage.py produces for smoother interoperability with various consumers of this data (e.g., the VSCode Coverage Gutters extension, Code Coverage for Bitbucket Server, etc.).

I have the constraint that coverage.py is invoked inside a temporary working directory (specifically, a bazel test sandbox) that does not contain the package to be measured. Rather, the package to be measured is installed in some virtualenv's site-packages directory.

I tried using coverage.py's path remapping with relative_files = true to produce relative paths in coverage.py's output, but because coverage.py's path remapping is sensitive to the current working directory, and the package to be measured is not inside the current working directory, this is not working.

It seems this could be fixed by allowing the base directory for source resolution to be explicitly configured (defaulting to $PWD if not provided), and/or adding support for an additional, final path remapping step that applies configured substitutions without first checking for file existence.

Reproduction steps:

~
❯ cd $(mktemp -d)

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ vim pyproject.toml

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ cat pyproject.toml
[tool.coverage.run]
source_pkgs = ["demo"]
relative_files = true

[tool.coverage.paths]
source = [
  "demo",
  "/**/demo",
]

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ mkdir demo

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ vim demo/__init__.py

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ cat demo/__init__.py
def foo():
    return 42

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ vim test_demo.py

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ cat test_demo.py
from unittest import TestCase, main

from demo import foo


class TestFoo(TestCase):
    def test_foo(self):
        self.assertEqual(foo(), 42)


if __name__ == "__main__":
    main()


xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ python3 -m venv .venv

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ .venv/bin/pip install -q coverage[toml]

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ set -gx SRC_DIR $(pwd)


xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR
❯ cd $(mktemp -d)

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hM
❯ $SRC_DIR/.venv/bin/coverage run --rcfile=$SRC_DIR/pyproject.toml $SRC_DIR/test_demo.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hM
❯ $SRC_DIR/.venv/bin/coverage lcov --rcfile=$SRC_DIR/pyproject.toml --debug=pathmap  # no luck:
Aliases (relative=True):
 Rule: '/**/demo' -> 'demo/' using regex '[/\\\\](.*[/\\\\])?demo[/\\\\]'
Rule '/**/demo' changed '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/__init__.py' to 'demo/__init__.py' which doesn't exist, continuing
No rules match, path '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/__init__.py' is unchanged
Wrote LCOV report to coverage.lcov

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hM
❯ vim $SRC_DIR/pyproject.toml  # try to fix by making the first path in "paths" absolute, so coverage.py can resolve it independently of $PWD:

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hM
❯ cat $SRC_DIR/pyproject.toml
[tool.coverage.run]
source_pkgs = ["demo"]
relative_files = true

[tool.coverage.paths]
source = [
  # Make this first path absolute:
  "${SRC_DIR-.}/demo",
  "/**/demo",
]

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hM
❯ $SRC_DIR/.venv/bin/coverage lcov --rcfile=$SRC_DIR/pyproject.toml --debug=pathmap  # Coverage now resolves the path and applies the transformation, but the transformation now includes the absolute path, so we haven't gotten anywhere:
Aliases (relative=True):
 Rule: '/**/demo' -> '/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/' using regex '[/\\\\](.*[/\\\\])?demo[/\\\\]'
Matched path '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/__init__.py' to rule '/**/demo' -> '/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/', producing '/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/__init__.py'
Wrote LCOV report to coverage.lcov

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hM
❯ cat coverage.lcov  #  paths are absolute:
TN:
SF:/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/__init__.py
DA:1,1,8W6LTH60TaV5ZrrD7DkxQQ
DA:2,1,KjVIXJCtSHJ6Jnfwt67A8Q
LF:2
LH:2
end_of_record

Perhaps some new setting like relative_to could complement relative_paths = true to enable this use case?

[nedbat]

I haven't tried to reproduce this, but: the [paths] setting is only used by the coverage combine command. Can you try using that command to process the data files? Also "/**/demo" I think can just be "*/demo".

[jab]

Thanks very much for taking a look at this, @nedbat.

I went through the reproduction steps above, but this time changed /**/demo to */demo and added --parallel-mode to the coverage run invocation (which I think is necessary for coverage combine). I'm still getting absolute paths with coverage combine, and similar "doesn't exist...No rules match" output.

click for repro using "coverage combine"

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnO
❯ ls -a
./  ../

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnO
❯ $SRC_DIR/.venv/bin/coverage run --parallel-mode --rcfile=$SRC_DIR/pyproject.toml $SRC_DIR/test_demo.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnO
❯ ls -a
./                             ../                            .coverage.jabmbp.19368.160174

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnO
❯ $SRC_DIR/.venv/bin/coverage combine --rcfile=$SRC_DIR/pyproject.toml --debug=pathmap
Aliases (relative=True):
 Rule: '*/demo' -> 'demo/' using regex '(.*[/\\\\])?demo[/\\\\]'
Rule '*/demo' changed '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py' to 'demo/__init__.py' which doesn't exist, continuing
No rules match, path '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py' is unchanged
Combined data file .coverage.jabmbp.19368.160174

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnO
❯ ls -a
./         ../        .coverage

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnO
❯ $SRC_DIR/.venv/bin/coverage lcov --rcfile=$SRC_DIR/pyproject.toml --debug=pathmap
Aliases (relative=True):
 Rule: '*/demo' -> 'demo/' using regex '(.*[/\\\\])?demo[/\\\\]'
Rule '*/demo' changed '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py' to 'demo/__init__.py' which doesn't exist, continuing
No rules match, path '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py' is unchanged
Wrote LCOV report to coverage.lcov

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnO
❯ cat coverage.lcov
TN:
SF:/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py
DA:1,1,8W6LTH60TaV5ZrrD7DkxQQ
DA:2,1,KjVIXJCtSHJ6Jnfwt67A8Q
LF:2
LH:2
end_of_record

Please let me know if there's anything further I can provide.

[jab]

Note, I see the same --debug=pathmap output from coverage combine as I do from lcov, which made me think that there's some shared $PWD-sensitive logic when resolving paths.

[nedbat]

Bug #1752 was recently fixed, but it doesn't improve this situation. Here the problem is this line:

Rule '*/demo' changed '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py' to 'demo/__init__.py' which doesn't exist, continuing

The important part is which doesn't exist: the relative path has to be relative to the current directory when the reporting happens.

[nedbat]

I'm going to close this: relative filenames are file names, not module names. I'm not sure how coverage could determine the relative file name to a file in a completely different directory.

[jab]

Perhaps some new setting like relative_to could complement relative_paths = true to enable this use case?

Would this be worth considering?

Note, I’m not currently affected by this, but expect to be next time I have to use coverage.py with Bazel.

[jab]

(and/or support applying arbitrary transformations to paths in a final post-processing step)

[nedbat]

Perhaps some new setting like relative_to could complement relative_paths = true to enable this use case?

Would this be worth considering?

I don't use Bazel, so I don't understand the changing of directories. Instead of a relative_to, could you use an environment variable to add a path to the [path] section that would do the right thing?

(BTW: what are you quoting from?)

[jab]

(I was quoting myself in the last line of this issue's description after you expand "reproduction steps". Will move that outside the <details> block now.)

Last time I had to do this I was using an older version of Bazel with Python rules that didn't support this out-of-the-box (so we had to roll our own and hit this issue). Now I see https://bazel.build/configure/coverage#python links to https://github.com/bazelbuild/rules_python/blob/main/docs/sphinx/coverage.md which says there is now builtin coverage support via coverage.py, so I think the next step here is to see if this is still an issue with that.

If you want to close this in the meantime, please feel free, though if the idea of supporting better path remapping/truncating would be useful for addressing other issues, feel free to use this to track that if it makes sense.

[jab]

Tried to repro this using the latest stable version of https://github.com/bazelbuild/rules_python, but its coverage implementation does not currently support custom coverage.py configs :(