feat(toolchain): Extend Python Testing Toolchain with COVERAGE_RC Support #2246

ewianda · 2024-09-24T21:19:52Z

This PR enhances the Test Runner environment to support custom coverage configuration via the COVERAGE_RC environment variable. Inspired by this approach, this feature allows users to specify a .coveragerc file in any compatible format to customize coverage reporting.

Key Changes
Adds support for the COVERAGE_RC environment variable in the Test Runner environment.
Enables users to define and pass their own .coveragerc configurations, enhancing flexibility for various testing and reporting requirements.

close #1434

examples/bzlmod/BUILD.bazel

aignas

This is certainly interesting. It seems that there are //tests/python/python_tests.bzl failing due to the mock module_ctx not being updated.

Shall we mark this as Ready for review and ask others to also have a look? Before doing that could you please update the PR description?

examples/bzlmod/BUILD.bazel

examples/bzlmod/MODULE.bazel

ewianda · 2024-10-26T15:48:03Z

This is certainly interesting. It seems that there are //tests/python/python_tests.bzl failing due to the mock module_ctx not being updated.

Shall we mark this as Ready for review and ask others to also have a look? Before doing that could you please update the PR description?

I am going to work on the description and get the tests to pass, then we can open it up for review.

aignas

Added some extra thoughts about the API design

python/private/py_test_toolchain.bzl

python/private/python.bzl

examples/bzlmod/tests/BUILD.bazel

python/private/py_test_toolchain.bzl

python/private/python_test.bzl

examples/bzlmod/tests/BUILD.bazel

ewianda · 2024-10-31T03:51:02Z

The test under Bazel 6.4.0 test is failing. My guess is because this native java py_test, If that is the case is the a mechanism to run the test conditionally

aignas · 2024-11-01T14:09:01Z

Ah, this is because bazel 6.4 is not using the starlark implementation of the rules, so it is expected to not work for now.

examples/bzlmod/tests/BUILD.bazel

aignas · 2024-11-04T01:57:23Z

Idea for why the Windows build is failing - it is using a very different bootstrap mechanism and it could be that the coveragerc is not wired properly. I am not 100% sure how we should do this. @rickeylev, any thoughts here?

ewianda · 2024-11-05T14:55:27Z

@rickeylev @aignas I have tried to figure out the issue with windows with no success. I noticed that C:\Users\ELVISW~1\AppData\Local\Temp\Bazel.runfiles__dysvgw6\runfiles\_main\.coveragerc is missing C:\Users\ElvisWianda\Documents\rules_python\examples\bzlmod\bazel-bin\tests\coverage_rc_is_set_test.exe.runfiles\_main/.coveragerc
exists

Figured out the issue

ewianda · 2024-12-07T19:24:19Z

ping @aignas @rickeylev

aignas

I like the general direction, but my main question is about how we could implement the additions of the toolchain so that we could have different flavours of py_test.

examples/bzlmod/tests/BUILD.bazel

CHANGELOG.md

python/config_settings/transition.bzl

aignas · 2024-12-09T07:54:27Z

python/extensions/python_test.bzl

+use_repo(python_test, "py_test_toolchain")
+register_toolchains("@py_test_toolchain//:all")


The python_test extension in theory could register to toolchain itself, just like python does it. What do you think?

aignas · 2024-12-09T07:56:15Z

python/private/py_executable.bzl

            if effective_runtime.coverage_files:
                transitive.append(effective_runtime.coverage_files)
+        if is_test:
+            py_test_toolchain = ctx.exec_groups["test"].toolchains[PY_TEST_TOOLCHAIN_TYPE]


I think this should be something that is a parameter to this function. I am thinking that we could somehow allow users creating executable rules with their own toolchain types. I would love to be able to move toward the direction of #1647.

I think we should use the ability to use a callback macro at the level of the toolchain. I didn't use that approach here since I was not adding an executable. Will change the implementation.

aignas · 2024-12-09T07:57:18Z

python/private/py_executable.bzl

+            py_test_toolchain = ctx.exec_groups["test"].toolchains[PY_TEST_TOOLCHAIN_TYPE]
+            if py_test_toolchain:
+                coverage_rc = py_test_toolchain.py_test_info.coverage_rc
+                extra_test_env = {"COVERAGE_RC": coverage_rc.files.to_list()[0].short_path}


I think in general we should check the ctx.configuration.coverage_enabled and add this in only then.

Regarding my comment above, if we would like to generalize it, do you have ideas how we could do it?

aignas · 2024-12-09T08:05:07Z

python/private/py_test_rule.bzl

+    exec_groups = {
+        "test": exec_group(toolchains = [config_common.toolchain_type(PY_TEST_TOOLCHAIN_TYPE, mandatory = False)]),
+    },


Ideally I would love to be able to express this as

Suggested change

exec_groups = {

"test": exec_group(toolchains = [config_common.toolchain_type(PY_TEST_TOOLCHAIN_TYPE, mandatory = False)]),

},

extra_toolchains = [PY_TEST_TOOLCHAIN_TYPE],

extend_foo = _my_function_to_add_env_and_set_coverage,

exec_groups = {

"test": exec_group(toolchains = [config_common.toolchain_type(PY_TEST_TOOLCHAIN_TYPE, mandatory = False)]),

},

My goal is for us to extend py_test with toolchain such that it would be possible to define a different flavour of py_test with its own toolchain and for that we would be reusing the create_executable_rule snippet like above.

Do you have any ideas?

Based on the C++ Testing Toolchain approach, it is better to have this configured at the toolchain level rather than the rule level. If I understand this correctly, users will be required to specify extend_foo on every py_test rule

I think I don't understand how _my_function_to_add_env_and_set_coverage will access the information provided by the toolchain.

Inspired by https://github.com/trybka/scraps/blob/master/cc_test.md This PR extends Test Runner enviroment to provide a coveragerc enviroment variable COVERAGE_RC, allowing user to provide coverage resource in what ever format

Create partially-bound function for configuring py_test

aignas

In general I think I like the approach here and would like to hear what @rickeylev thinks about such a way to do dependency injection.

I can see how I could add a @pypi//pytest_bazel target in the python_test.configure call and in this way tell bazel that we have to use that package for running all of the tests. I would probably then need to somehow add the extra invocation of python -m pytest_bazel somewhere in the py_executable.bzl or related in order to benefit from the extra dependency in there.

Remaining bits before merging:

Have more than just me believing in fruitfulness of this approach.
Add this to the //docs:BUILD.bazel and add an EXPERIMENTAL note in the symbols.
Profit?

aignas · 2025-01-26T08:02:17Z

python/private/py_test_toolchain.bzl

+                    func = _get_runner,
+                    args = {
+                        "coverage_rc": ctx.attr.coverage_rc,
+                    },


nit: I think we could use a lambda here so that the call site can just do get_runner(my_extra_args).

aignas · 2025-01-26T08:05:56Z

python/private/python_test.bzl

+            register_py_test_toolchain(
+                name = "py_test_toolchain",
+                coverage_rc = tag.coveragerc,
+                register_toolchains = False,


I see that the register_toolchains arg here has been borrowed from the uv toolchain. I am personally not a fan for the func name to say register_py_test_toolchain and then there being an arg register_toolchains = False. So what are we doing here? Maybe just not registering the toolchains and calling the function py_test_toolchain_repositories would be more acurate?

aignas · 2025-01-26T08:06:22Z

python/private/python_test.bzl

+    """
+    for mod in module_ctx.modules:
+        for tag in mod.tags.configure:
+            register_py_test_toolchain(


Should we allow non-root modules to do the registrations? I think we could just ignore any non-root modules.

ewianda · 2025-01-30T14:56:31Z

In general I think I like the approach here and would like to hear what @rickeylev thinks about such a way to do dependency injection.

I can see how I could add a @pypi//pytest_bazel target in the python_test.configure call and in this way tell bazel that we have to use that package for running all of the tests. I would probably then need to somehow add the extra invocation of python -m pytest_bazel somewhere in the py_executable.bzl or related in order to benefit from the extra dependency in there.

Remaining bits before merging:

Have more than just me believing in fruitfulness of this approach.

Add this to the //docs:BUILD.bazel and add an EXPERIMENTAL note in the symbols.

Profit?

@rickeylev any chance to take a look, I work on your suggestions @aignas

ewianda · 2025-08-15T14:09:49Z

@aignas is this still something to pursue? I want to upgrade rules_python, and find myself needing to carry forward a patch for this.

aignas · 2025-08-20T09:28:09Z

I'm torn about this. On one hand I don't think we have the coverage story sorted out properly, but I think the proposed PR may be a little bit hard to maintain. There was no consensus among the maintainers about maintaing this sort of API long term.

There was some desire to make this a user space pnoblem - like what pytest-bazel is doing, but there is a need for more integration. Maybe there can be a way in between.

ewianda · 2025-08-20T20:49:03Z

I'm torn about this. On one hand I don't think we have the coverage story sorted out properly, but I think the proposed PR may be a little bit hard to maintain. There was no consensus among the maintainers about maintaing this sort of API long term.

There was some desire to make this a user space pnoblem - like what pytest-bazel is doing, but there is a need for more integration. Maybe there can be a way in between.

Fair enough, I had proposed a low maintenance approach (open for debate) of just providing the coveragerc using an environemnt variable. This is how I am currently solving this as illustrated here . This should not be far out, given that we already use the PYTHON_COVERAGE env variable to override the coverage tool. Thoughts

This approach was discused in #1434

and implemented in bazelbuild/bazel#19656

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch from c7a142b to 173853e Compare September 24, 2024 21:20

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch 4 times, most recently from 4c0451e to 5a16df4 Compare October 22, 2024 19:02

ewianda commented Oct 22, 2024

View reviewed changes

examples/bzlmod/BUILD.bazel Show resolved Hide resolved

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch from 5a16df4 to 039012f Compare October 22, 2024 19:11

aignas reviewed Oct 26, 2024

View reviewed changes

examples/bzlmod/BUILD.bazel Show resolved Hide resolved

examples/bzlmod/MODULE.bazel Outdated Show resolved Hide resolved

aignas reviewed Oct 27, 2024

View reviewed changes

python/private/py_test_toolchain.bzl Outdated Show resolved Hide resolved

python/private/python.bzl Outdated Show resolved Hide resolved

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch from 039012f to 0d3a405 Compare October 30, 2024 06:19

ewianda changed the title ~~Feat configure coverage rc env variable~~ feat(toolchain): Extend Python Testing Toolchain with COVERAGE_RC Support Oct 30, 2024

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch 2 times, most recently from f256992 to 9ebd564 Compare October 30, 2024 13:35

ewianda commented Oct 30, 2024

View reviewed changes

examples/bzlmod/tests/BUILD.bazel Show resolved Hide resolved

ewianda commented Oct 30, 2024

View reviewed changes

python/private/py_test_toolchain.bzl Show resolved Hide resolved

aignas reviewed Oct 31, 2024

View reviewed changes

python/private/py_test_toolchain.bzl Outdated Show resolved Hide resolved

python/private/python_test.bzl Show resolved Hide resolved

examples/bzlmod/tests/BUILD.bazel Show resolved Hide resolved

ewianda mentioned this pull request Oct 31, 2024

feat(toolchain) Add coveragepy configuration attribute #2224

Closed

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch 2 times, most recently from 02d40e0 to 9e4665c Compare October 31, 2024 15:56

aignas reviewed Nov 1, 2024

View reviewed changes

examples/bzlmod/tests/BUILD.bazel Show resolved Hide resolved

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch 2 times, most recently from 0371be3 to 3060b83 Compare November 1, 2024 21:16

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch from de0204d to d7181bc Compare November 5, 2024 18:44

ewianda marked this pull request as ready for review November 5, 2024 18:53

ewianda requested a review from rickeylev as a code owner November 5, 2024 18:53

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch from d7181bc to 2270aa0 Compare November 5, 2024 20:03

ewianda requested a review from aignas November 5, 2024 20:03

aignas reviewed Dec 10, 2024

View reviewed changes

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch from 2270aa0 to acbd8c7 Compare December 14, 2024 19:08

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch from acbd8c7 to e670fd3 Compare January 22, 2025 06:06

ewianda requested a review from groodt as a code owner January 22, 2025 06:06

ewianda added 3 commits January 22, 2025 01:10

feat(toolchain): Python testing toolchain

2667f78

Inspired by https://github.com/trybka/scraps/blob/master/cc_test.md This PR extends Test Runner enviroment to provide a coveragerc enviroment variable COVERAGE_RC, allowing user to provide coverage resource in what ever format

Add env variable to rule transitions

c3f1d7a

Move toolchain logic into get runtime details

c111829

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch from e670fd3 to 23011c4 Compare January 22, 2025 06:13

Factor out Default and RuntimeInfo providers

214972f

Create partially-bound function for configuring py_test

ewianda force-pushed the feat-configure-coverage-rc-env-variable branch from 23011c4 to 214972f Compare January 22, 2025 06:15

aignas reviewed Jan 26, 2025

View reviewed changes

aignas mentioned this pull request Feb 3, 2025

fix(coverage): prevent issues when packages execute files in tmp dir. #2599

Closed

		use_repo(python_test, "py_test_toolchain")
		register_toolchains("@py_test_toolchain//:all")

Uh oh!

Uh oh!

feat(toolchain): Extend Python Testing Toolchain with COVERAGE_RC Support #2246

Are you sure you want to change the base?

feat(toolchain): Extend Python Testing Toolchain with COVERAGE_RC Support #2246

Uh oh!

Conversation

ewianda commented Sep 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

aignas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ewianda commented Oct 26, 2024

Uh oh!

aignas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ewianda commented Oct 31, 2024

Uh oh!

aignas commented Nov 1, 2024

Uh oh!

Uh oh!

aignas commented Nov 4, 2024

Uh oh!

ewianda commented Nov 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ewianda commented Dec 7, 2024

Uh oh!

aignas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ewianda Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aignas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ewianda commented Jan 30, 2025

Uh oh!

ewianda commented Aug 15, 2025

Uh oh!

aignas commented Aug 20, 2025

Uh oh!

ewianda commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

ewianda commented Sep 24, 2024 •

edited

Loading

ewianda commented Nov 5, 2024 •

edited

Loading

ewianda Dec 12, 2024 •

edited

Loading

ewianda commented Aug 20, 2025 •

edited

Loading