fix: don't require system Python to perform bootstrapping

CHANGELOG.md

@@ -1,3 +1,6 @@
+:::{default-domain} bzl
+:::
+
 # rules_python Changelog
 
 This is a human-friendly changelog in a keepachangelog.com style format.
@@ -31,7 +34,7 @@ A brief description of the categories of changes:
   marked as `reproducible` and will not include any lock file entries from now
   on.
 
-* (gazelle): Remove gazelle plugin's python deps and make it hermetic. 
+* (gazelle): Remove gazelle plugin's python deps and make it hermetic.
   Introduced a new Go-based helper leveraging tree-sitter for syntax analysis.
   Implemented the use of `pypi/stdlib-list` for standard library module verification.
 
@@ -80,6 +83,16 @@ A brief description of the categories of changes:
   invalid usage previously but we were not failing the build. From now on this
   is explicitly disallowed.
 * (toolchains) Added riscv64 platform definition for python toolchains.
+* (rules) A new bootstrap implementation that doesn't require a system Python
+  is available. It can be enabled by setting
+  {obj}`--@rules_python//python:config_settings:bootstrap_impl=two_phase`. It
+  will become the default in a subsequent release.
+  ([#691](https://github.com/bazelbuild/rules_python/issues/691))
+* (providers) `PyRuntimeInfo` has two new attributes:
+  {obj}`PyRuntimeInfo.stage2_bootstrap_template` and
+  {obj}`PyRuntimeInfo.zip_main_template`.
+* (toolchains) A replacement for the Bazel-builtn autodetecting toolchain is
+  available. The `//python:autodetecting_toolchain` alias now uses it.
 
 [precompile-docs]: /precompiling
 

CONTRIBUTING.md

@@ -175,6 +175,7 @@ Issues should be triaged as follows:
   functionality, should also be filed in this repository but without the
   `core-rules` label.
 
+(breaking-changes)=
 ## Breaking Changes
 
 Breaking changes are generally permitted, but we follow a 3-step process for

docs/sphinx/api/python/config_settings/index.md

@@ -1,3 +1,5 @@
+:::{default-domain} bzl
+:::
 :::{bzl:currentfile} //python/config_settings:BUILD.bazel
 :::
 
@@ -66,3 +68,32 @@ Values:
 * `include_pyc`: Include `PyInfo.transitive_pyc_files` as part of the binary.
 * `disabled`: Don't include `PyInfo.transitive_pyc_files` as part of the binary.
 :::
+
+::::{bzl:flag} bootstrap_impl
+Determine how programs implement their startup process.
+
+Values:
+* `system_python`: Use a bootstrap that requires a system Python available
+  in order to start programs. This requires
+  {obj}`PyRuntimeInfo.bootstrap_template` to be a Python program.
+* `script`: Use a bootstrap that uses an arbitrary executable script (usually a
+  shell script) instead of requiring it be a Python program.
+
+:::{note}
+The `script` bootstrap requires the toolchain to provide the `PyRuntimeInfo`
+provider from `rules_python`. This loosely translates to using Bazel 7+ with a
+toolchain created by rules_python. Most notably, WORKSPACE builds default to
+using a legacy toolchain built into Bazel itself which doesn't support the
+script bootstrap. If not available, the `system_python` bootstrap will be used
+instead.
+:::
+
+:::{seealso}
+{obj}`PyRuntimeInfo.bootstrap_template` and
+{obj}`PyRuntimeInfo.stage2_bootstrap_template`
+:::
+
+:::{versionadded} 0.33.0
+:::
+
+::::

docs/sphinx/api/python/index.md

@@ -1,3 +1,5 @@
+:::{default-domain} bzl
+:::
 :::{bzl:currentfile} //python:BUILD.bazel
 :::
 
@@ -21,3 +23,21 @@ provides:
 * `PyRuntimeInfo`: The consuming target's target toolchain information
 
 :::
+
+::::{target} autodetecting_toolchain
+
+A simple toolchain that simply uses `python3` from the runtime environment.
+
+Note that this toolchain provides no build-time information, which makes it of
+limited utility.
+
+This is only provided to aid migration off the builtin Bazel toolchain 
+(`@bazel_tools//python:autodetecting_toolchain`), and is largely only applicable
+to WORKSPACE builds.
+
+:::{deprecated} unspecified
+
+Switch to using a hermetic toolchain or manual toolchain configuration instead.
+:::
+
+::::

docs/sphinx/bazel_inventory.txt

@@ -10,7 +10,7 @@ bool bzl:type 1 rules/lib/bool -
 int bzl:type 1 rules/lib/int -
 depset bzl:type 1 rules/lib/depset -
 dict bzl:type 1 rules/lib/dict -
-label bzl:doc 1 concepts/labels -
+label bzl:type 1 concepts/labels -
 attr.bool bzl:type 1 rules/lib/toplevel/attr#bool -
 attr.int bzl:type 1 rules/lib/toplevel/attr#int -
 attr.label bzl:type 1 rules/lib/toplevel/attr#label -
@@ -21,6 +21,7 @@ list bzl:type 1 rules/lib/list -
 python bzl:doc 1 reference/be/python -
 str bzl:type 1 rules/lib/string -
 struct bzl:type 1 rules/lib/builtins/struct -
-target-name bzl:doc 1 concepts/labels#target-names -
+Name bzl:type 1 concepts/labels#target-names -
 CcInfo bzl:provider 1 rules/lib/providers/CcInfo -
 CcInfo.linking_context bzl:provider-field 1 rules/lib/providers/CcInfo#linking_context -
+ToolchainInfo bzl:type 1 rules/lib/providers/ToolchainInfo.html -

docs/sphinx/pip.md

@@ -150,7 +150,7 @@ ARG=$1  # but we don't do anything with it as it's always "get"
 # formatting is optional
 echo '{'
 echo '  "headers": {'
-echo '    "Authorization": ["Basic dGVzdDoxMjPCow=="]
+echo '    "Authorization": ["Basic dGVzdDoxMjPCow=="]'
 echo '  }'
 echo '}'
 ```

docs/sphinx/support.md

@@ -46,7 +46,8 @@ incremental fashion.
 
 Breaking changes are allowed, but follow a process to introduce them over
 a series of releases to so users can still incrementally upgrade. See the
-[Breaking Changes](contributing#breaking-changes) doc for the process.
+[Breaking Changes](#breaking-changes) doc for the process.
+
 
 ## Experimental Features
 

docs/sphinx/toolchains.md

@@ -1,3 +1,6 @@
+:::{default-domain} bzl
+:::
+
 # Configuring Python toolchains and runtimes
 
 This documents how to configure the Python toolchain and runtimes for different
@@ -193,7 +196,7 @@ load("@rules_python//python:repositories.bzl", "py_repositories")
 py_repositories()
 ```
 
-#### Workspace toolchain registration
+### Workspace toolchain registration
 
 To register a hermetic Python toolchain rather than rely on a system-installed interpreter for runtime execution, you can add to the `WORKSPACE` file:
 
@@ -221,3 +224,21 @@ pip_parse(
 After registration, your Python targets will use the toolchain's interpreter during execution, but a system-installed interpreter
 is still used to 'bootstrap' Python targets (see https://github.com/bazelbuild/rules_python/issues/691).
 You may also find some quirks while using this toolchain. Please refer to [python-build-standalone documentation's _Quirks_ section](https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html).
+
+## Autodetecting toolchain
+
+The autodetecting toolchain is a deprecated toolchain that is built into Bazel.
+It's name is a bit misleading: it doesn't autodetect anything. All it does is
+use `python3` from the environment a binary runs within. This provides extremely
+limited functionality to the rules (at build time, nothing is knowable about
+the Python runtime).
+
+Bazel itself automatically registers `@bazel_tools//python:autodetecting_toolchain`
+as the lowest priority toolchain. For WORKSPACE builds, if no other toolchain
+is registered, that toolchain will be used. For bzlmod builds, rules_python
+automatically registers a higher-priority toolchain; it won't be used unless
+there is a toolchain misconfiguration somewhere.
+
+To aid migration off the Bazel-builtin toolchain, rules_python provides
+{obj}`@rules_python//python:autodetecting_toolchain`. This is an equivalent
+toolchain, but is implemented using rules_python's objects.

examples/bzlmod/test.py

@@ -14,6 +14,7 @@
 
 import os
 import pathlib
+import re
 import sys
 import unittest
 
@@ -63,16 +64,47 @@ def test_coverage_sys_path(self):
             first_item.endswith("coverage"),
             f"Expected the first item in sys.path '{first_item}' to not be related to coverage",
         )
+
+        # We're trying to make sure that the coverage library added by the
+        # toolchain is _after_ any user-provided dependencies. This lets users
+        # override what coverage version they're using.
+        first_coverage_index = None
+        last_user_dep_index = None
+        for i, path in enumerate(sys.path):
+            if re.search("rules_python.*~pip~", path):
+                last_user_dep_index = i
+            if first_coverage_index is None and re.search(
+                ".*rules_python.*~python~.*coverage.*", path
+            ):
+                first_coverage_index = i
+
         if os.environ.get("COVERAGE_MANIFEST"):
+            self.assertIsNotNone(
+                first_coverage_index,
+                "Expected to find toolchain coverage, but "
+                + f"it was not found.\nsys.path:\n{all_paths}",
+            )
+            self.assertIsNotNone(
+                first_coverage_index,
+                "Expected to find at least one uiser dep, "
+                + "but none were found.\nsys.path:\n{all_paths}",
+            )
             # we are running under the 'bazel coverage :test'
-            self.assertTrue(
-                "_coverage" in last_item,
-                f"Expected {last_item} to be related to coverage",
+            self.assertGreater(
+                first_coverage_index,
+                last_user_dep_index,
+                "Expected coverage provided by the toolchain to be after "
+                + "user provided dependencies.\n"
+                + f"Found coverage at index: {first_coverage_index}\n"
+                + f"Last user dep at index: {last_user_dep_index}\n"
+                + f"Full sys.path:\n{all_paths}",
             )
-            self.assertEqual(pathlib.Path(last_item).name, "coverage")
         else:
-            self.assertFalse(
-                "coverage" in last_item, f"Expected coverage tooling to not be present"
+            self.assertIsNone(
+                first_coverage_index,
+                "Expected toolchain coverage to not be present\n"
+                + f"Found coverage at index: {first_coverage_index}\n"
+                + f"Full sys.path:\n{all_paths}",
             )
 
     def test_main(self):

python/BUILD.bazel

@@ -24,6 +24,7 @@ that @rules_python//python is only concerned with the core rules.
 """
 
 load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
+load("//python/private:autodetecting_toolchain.bzl", "define_autodetecting_toolchain")
 load(":current_py_toolchain.bzl", "current_py_toolchain")
 
 package(default_visibility = ["//visibility:public"])
@@ -318,14 +319,11 @@ toolchain_type(
 # safe if you know for a fact that your build is completely compatible with the
 # version of the `python` command installed on the target platform.
 
-alias(
-    name = "autodetecting_toolchain",
-    actual = "@bazel_tools//tools/python:autodetecting_toolchain",
-)
+define_autodetecting_toolchain(name = "autodetecting_toolchain")
 
 alias(
     name = "autodetecting_toolchain_nonstrict",
-    actual = "@bazel_tools//tools/python:autodetecting_toolchain_nonstrict",
+    actual = ":autodetecting_toolchain",
 )
 
 # ========= Packaging rules =========

python/config_settings/BUILD.bazel

@@ -1,6 +1,7 @@
 load("@bazel_skylib//rules:common_settings.bzl", "string_flag")
 load(
     "//python/private:flags.bzl",
+    "BootstrapImplFlag",
     "PrecompileAddToRunfilesFlag",
     "PrecompileFlag",
     "PrecompileSourceRetentionFlag",
@@ -52,3 +53,11 @@ string_flag(
     # NOTE: Only public because its an implicit dependency
     visibility = ["//visibility:public"],
 )
+
+string_flag(
+    name = "bootstrap_impl",
+    build_setting_default = BootstrapImplFlag.SYSTEM_PYTHON,
+    values = sorted(BootstrapImplFlag.__members__.values()),
+    # NOTE: Only public because its an implicit dependency
+    visibility = ["//visibility:public"],
+)

python/private/BUILD.bazel

@@ -376,9 +376,54 @@ exports_files(
     visibility = ["//visibility:public"],
 )
 
+filegroup(
+    name = "stage1_bootstrap_template",
+    srcs = ["stage1_bootstrap_template.sh"],
+    # Not actually public. Only public because it's an implicit dependency of
+    # py_runtime.
+    visibility = ["//visibility:public"],
+)
+
+filegroup(
+    name = "stage2_bootstrap_template",
+    srcs = ["stage2_bootstrap_template.py"],
+    # Not actually public. Only public because it's an implicit dependency of
+    # py_runtime.
+    visibility = ["//visibility:public"],
+)
+
+filegroup(
+    name = "zip_main_template",
+    srcs = ["zip_main_template.py"],
+    # Not actually public. Only public because it's an implicit dependency of
+    # py_runtime.
+    visibility = ["//visibility:public"],
+)
+
+# NOTE: Windows builds don't use this bootstrap. Instead, a native Windows
+# program locates some Python exe and runs `python.exe foo.zip` which
+# runs the __main__.py in the zip file.
+alias(
+    name = "bootstrap_template",
+    actual = select({
+        ":is_script_bootstrap_enabled": "stage1_bootstrap_template.sh",
+        "//conditions:default": "python_bootstrap_template.txt",
+    }),
+    # Not actually public. Only public because it's an implicit dependency of
+    # py_runtime.
+    visibility = ["//visibility:public"],
+)
+
 # Used to determine the use of `--stamp` in Starlark rules
 stamp_build_setting(name = "stamp")
 
+config_setting(
+    name = "is_script_bootstrap_enabled",
+    flag_values = {
+        "//python/config_settings:bootstrap_impl": "script",
+    },
+)
+
 print_toolchains_checksums(name = "print_toolchains_checksums")
 
 # Used for py_console_script_gen rule

python/private/autodetecting_toolchain.bzl

@@ -32,7 +32,7 @@ def define_autodetecting_toolchain(name):
     # buildifier: disable=native-py
     py_runtime(
         name = "_autodetecting_py3_runtime",
-        interpreter = ":py3wrapper.sh",
+        interpreter = "//python/private:autodetecting_toolchain_interpreter.sh",
         python_version = "PY3",
         stub_shebang = "#!/usr/bin/env python3",
         visibility = ["//visibility:private"],

python/private/common/common.bzl

@@ -182,7 +182,7 @@ def create_cc_details_struct(
         cc_toolchain = cc_toolchain,
     )
 
-def create_executable_result_struct(*, extra_files_to_build, output_groups):
+def create_executable_result_struct(*, extra_files_to_build, output_groups, extra_runfiles = None):
     """Creates a `CreateExecutableResult` struct.
 
     This is the return value type of the semantics create_executable function.
@@ -192,13 +192,15 @@ def create_executable_result_struct(*, extra_files_to_build, output_groups):
             included as default outputs.
         output_groups: dict[str, depset[File]]; additional output groups that
             should be returned.
+        extra_runfiles: A runfiles object of additional runfiles to include.
 
     Returns:
         A `CreateExecutableResult` struct.
     """
     return struct(
         extra_files_to_build = extra_files_to_build,
         output_groups = output_groups,
+        extra_runfiles = extra_runfiles,
     )
 
 def union_attrs(*attr_dicts, allow_none = False):