Reimplement rules in Starlark in rules_python #1069

rickeylev · 2023-02-14T07:05:45Z

This is the general tracking issue for re-implementing the Bazel-bundled Python rules in Starlark in the rules_python repo.

The Bazel-side issue is bazelbuild/bazel#15897

As of now (Feb 2023), the Bazel bundled rules (not providers or flags, though) have all been re-implemented in Starlark. They will used in Bazel 7. Hooks for low-level Bazel behaviors the Python rules need are exposed to the rules_python repo. So now this is mostly a matter of copying that code over here, cleaning it up, and flushing out any missing hooks.

TODO

Copy Bazel code over to rules_python
Re-implement Bazel tests in rules_python
Re-implement flags in Starlark
Setup a CI config that uses e.g. bazel=rolling to get access to pre-release bazel versions
Enable pystar by default

That's the high level gist, some lower level TODOs are:

Implement compatibility shim so rules_python and Bazel rules can interoperate to some degree
Deprecate and remove Bazel-bundled flags
Deprecate and remove Python-related parts in @bazel_tools

An optional bonus thing is to backport the low-level hooks to 5.x and/or 6.x. This makes them available sooner, so that we don't have to wait for Bazel 7. There isn't a ton to backport (I intentionally made them to be amenable to this idea), but there's several small pieces. The two general pieces are exposing the flags as fragments and the py_internal object:

py_internal object and it's backing pieces:
Configuration fields (Bazel builtin flags); Strictly speaking, only flags that don't have a fixed set of values (e.g arbitrary string or labels) need to be made available. Other flags can be "wrapped" in a regular Starlark-defined flag by just having an exhaustive listing of config_settings
- --python_path and --python_top from BazelPythonConfiguration; note, however, that --python_top is supposed to be unused (superceded by toolchains) and --python_path is probably unused (unclear; some comments indicate it might be used as part of windows support)
- --build_python_zip from PythonConfiguration.java; note, however, it, too, can be replicated in Starlark by using platforms or the py_internal.get_current_os_name() helper.

The text was updated successfully, but these errors were encountered:

This is a clean-copy without any changes to provide a baseline Work towards bazelbuild#1069

This is to provide some regression tests for the Starlark rewrite. Work towards bazelbuild#1069

This is to provide some regression tests for the Starlark rewrite. These tests are approximately the same as Bazel's Java-implemented tests. Work towards bazelbuild#1069

This is to provide some regression tests for the Starlark rewrite. These tests are approximately the same as Bazel's Java-implemented tests. Work towards #1069

This is so that the transitive dependencies of defs.bzl can be easily found and validated; some Google internal tooling does this validation. The old comment indicated bzl_library wasn't used to avoid a dependency on skylib, however, we've since added a dependency on skylib. Work towards bazelbuild#1069

This is so that the transitive dependencies of defs.bzl can be easily found and validated; some Google internal tooling does this validation. The old comment indicated bzl_library wasn't used to avoid a dependency on skylib, however, we've since added a dependency on skylib. Work towards #1069

This is basically a performance optimization. Bazel can only see that a bzl file changed, not the particular contents. This means that any downstream bzl file loading it is invalidated, even if it doesn't load any of the affected code. As an example, if a package only loads `py_library.bzl`, then changing `py_test.bzl` doesn't need to invalidate all libraries. Work towards bazelbuild#1069

This is basically a performance optimization. Bazel can only see that a bzl file changed, not the particular contents. This means that any downstream bzl file loading it is invalidated, even if it doesn't load any of the affected code. As an example, if a package only loads `py_library.bzl`, then changing `py_test.bzl` doesn't need to invalidate all libraries. * Also removes some more extraneous license comments Work towards bazelbuild#1069

This is basically a performance optimization. Bazel can only see that a bzl file changed, not the particular contents. This means that any downstream bzl file loading it is invalidated, even if it doesn't load any of the affected code. As an example, if a package only loads `py_library.bzl`, then changing `py_test.bzl` doesn't need to invalidate all libraries. * Also removes some more extraneous license comments Work towards #1069

This provides a public entry point for loading the underlying `PyCcLinkParamsProvider` provider that is built into Bazel. This provider isn't yet usable from Bazel, but adding a loadable way for it to migrate off the built-in rules is the first step. Work towards bazelbuild#1069

This provides a public entry point for loading the underlying `PyCcLinkParamsProvider` provider that is built into Bazel. This provider isn't yet usable from Bazel, but adding a loadable way for it to migrate off the built-in rules is the first step. Work towards #1069

This is a pristine copy of the Starlark implementation of the Python rules from Bazel. This code isn't loaded and won't work as-is. Modifications to make it work will be made is subsequent changes. Work towards bazelbuild#1069

This makes the rules_python Starlark implementation accept and return the builtin providers. This allows depending on, and being depended on by, the builtin rules, which enables the two rule sets to interoperate better. Work towards bazelbuild#1069

This makes the rules_python Starlark implementation accept and return the builtin providers. This allows depending on, and being depended on by, the builtin rules, which enables the two rule sets to interoperate better. Work towards #1069

This makes rules_python compatible with disabling using the native rules in Bazel. Work towards bazelbuild#1069

With the release of Bazel 7, support for 5 is no longer necessary. Work towards bazelbuild#1069

This makes rules_python compatible with disabling using the native rules in Bazel. While we're here, update some examples in docs to load py_runtime to better make clear the builtin py_runtime objects shouldn't be used. Work towards #1069

With the release of Bazel 7, support for 5 is no longer necessary. Also deletes all the integration tests of the examples. They were broken when run from the main repo, and CI was only building them, not running them, as part of the main repo due to having `--test_tag_filters=-integration-test`. CI runs them separately in their own workspace. Work towards #1069

Ryang20718 · 2023-12-29T18:30:05Z

wondering if pyc support is on the roadmap? #1761

… Bazel 7+ Respecting it for earlier versions was only done for testing purposes. The implementation requires the `py_internal` object, which only exists on Bazel 7 and higher. Work towards bazelbuild#1069

This makes the base rules use the Starlark code in rules_python ("pystar") instead of what is built into Bazel 7. The pystar implementation can be disabled by setting `RULES_PYTHON_ENABLE_PYSTAR=0` in your environment or using `--action_env`. Earlier versions of Bazel continue to use the rules built into Bazel regardless (the new implementation requires special APIs that only Bazel 7 has). Work towards bazelbuild#1069

… Bazel 7+ (#1698) Respecting it for earlier versions was only done for testing purposes. The implementation requires the `py_internal` object, which only exists on Bazel 7 and higher. Work towards #1069

This makes the base rules use the Starlark code in rules_python ("pystar") instead of what is built into Bazel 7. The pystar implementation can be disabled by setting `RULES_PYTHON_ENABLE_PYSTAR=0` in your environment or using `--action_env`. Earlier versions of Bazel continue to use the rules built into Bazel regardless (the new implementation requires special APIs that only Bazel 7 has). Work towards bazelbuild#1069

…7+ (#1699) This makes the base rules use the Starlark code in rules_python ("pystar") instead of what is built into Bazel 7. The pystar implementation can be disabled by setting `RULES_PYTHON_ENABLE_PYSTAR=0` in your environment or using `--action_env`. Earlier versions of Bazel continue to use the rules built into Bazel regardless (the new implementation requires special APIs that only Bazel 7 has). Work towards #1069

rickeylev · 2024-02-13T17:24:30Z

The 0.31.0 release of rules_python has enabled the rules_python Starlark implementation for Bazel 7+. 🎆 !

If you encounter issues, please file bugs in rules_python. I'm eager to identify and fix any issues. If you do encounter issues, setting RULES_PYTHON_ENABLE_PYSTAR=0 in your environment (directly or using --action_env) will disable the rules_python implementation.

The most likely problem to occur is using a combination of the Bazel-builtin rules/providers and the rules_python rules/providers. They should interoperate OK, but please report any issues.

Related: bazelbuild/rules_python#1069 (comment)

aignas · 2024-06-17T23:47:48Z

Let's close this as complete as it is working well. Thank you @rickeylev!

rickeylev added type: feature request core-rules Issues concerning the stubs for native rules labels Feb 14, 2023

rickeylev self-assigned this Feb 14, 2023

rickeylev added a commit to rickeylev/rules_python that referenced this issue Feb 24, 2023

Copy Bazel's Starlark-based Python rules to rules_python.

0bad1ce

This is a clean-copy without any changes to provide a baseline Work towards bazelbuild#1069

rickeylev added a commit to rickeylev/rules_python that referenced this issue Feb 27, 2023

test(core): Add analysis tests for base Python rules.

2f1da4e

This is to provide some regression tests for the Starlark rewrite. Work towards bazelbuild#1069

rickeylev added a commit to rickeylev/rules_python that referenced this issue Feb 27, 2023

test(core): Add analysis tests for base Python rules.

fa9261e

This is to provide some regression tests for the Starlark rewrite. Work towards bazelbuild#1069

rickeylev mentioned this issue Feb 27, 2023

test(core): Add analysis tests for base Python rules. #1102

Merged

rickeylev mentioned this issue Mar 8, 2023

feat: add bzl_library for defs.bzl and its dependencies #1115

Merged

rickeylev mentioned this issue Mar 9, 2023

feat: bzl file per rule/provider #1122

Merged

rickeylev pinned this issue Mar 10, 2023

rickeylev mentioned this issue Mar 10, 2023

feat: add public entry point for PyCcLinkParamsInfo #1128

Merged

rickeylev added a commit to rickeylev/rules_python that referenced this issue Dec 12, 2023

chore: use rules_python's py_runtime in generated toolchains

1a48a42

This makes rules_python compatible with disabling using the native rules in Bazel. Work towards bazelbuild#1069

rickeylev mentioned this issue Dec 12, 2023

fix: use rules_python's py_runtime in generated toolchains #1604

Merged

rickeylev added a commit to rickeylev/rules_python that referenced this issue Dec 14, 2023

chore: use rules_python's py_runtime in generated toolchains

f79ecad

This makes rules_python compatible with disabling using the native rules in Bazel. Work towards bazelbuild#1069

rickeylev added a commit to rickeylev/rules_python that referenced this issue Dec 14, 2023

chore: Update Bazel support: drop 5, minimum 6.2, current 7.0

5d02131

With the release of Bazel 7, support for 5 is no longer necessary. Work towards bazelbuild#1069

rickeylev mentioned this issue Dec 14, 2023

chore: Update Bazel support: drop 5, minimum 6.2, current 7.0 #1613

Merged

rickeylev mentioned this issue Jan 17, 2024

internal(pystar): Only respect RULES_PYTHON_ENABLE_PYSTAR env var for Bazel 7+ #1698

Merged

rickeylev mentioned this issue Jan 17, 2024

feat: enable rules_python Starlark implemention by default for Bazel 7+ #1699

Merged

junyer mentioned this issue Mar 12, 2024

How to set python interpreter within cc_binary pybind/pybind11_bazel#77

Closed

oxidase mentioned this issue Jun 1, 2024

[Bug]: aspect_rules_py is incompatible with rules_poetry aspect-build/rules_py#338

Closed

oxidase added a commit to oxidase/ofiuco that referenced this issue Jun 1, 2024

fix: use PyInfo from rules_package, fixes #60

bbf02b7

Related: bazelbuild/rules_python#1069 (comment)

oxidase added a commit to oxidase/ofiuco that referenced this issue Jun 1, 2024

fix: use PyInfo from rules_package, fixes #60

0ada6ff

Related: bazelbuild/rules_python#1069 (comment)

oxidase added a commit to oxidase/ofiuco that referenced this issue Jun 1, 2024

fix: use PyInfo from rules_package

14edcc3

Related: bazelbuild/rules_python#1069 (comment)

oxidase added a commit to oxidase/ofiuco that referenced this issue Jun 1, 2024

fix: use PyInfo from rules_package

5dd60b2

Related: bazelbuild/rules_python#1069 (comment)

aignas closed this as completed Jun 17, 2024

aignas unpinned this issue Jun 18, 2024

alexeagle mentioned this issue Oct 3, 2024

[Bug]: interpreter version error when defining a Python toolchain aspect-build/rules_py#365

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reimplement rules in Starlark in rules_python #1069

Reimplement rules in Starlark in rules_python #1069

rickeylev commented Feb 14, 2023 •

edited

Loading

Ryang20718 commented Dec 29, 2023

rickeylev commented Feb 13, 2024 •

edited

Loading

aignas commented Jun 17, 2024

Reimplement rules in Starlark in rules_python #1069

Reimplement rules in Starlark in rules_python #1069

Comments

rickeylev commented Feb 14, 2023 • edited Loading

Ryang20718 commented Dec 29, 2023

rickeylev commented Feb 13, 2024 • edited Loading

aignas commented Jun 17, 2024

rickeylev commented Feb 14, 2023 •

edited

Loading

rickeylev commented Feb 13, 2024 •

edited

Loading