Added arg for free threading support

[vfdev-5]

Description:

• Added free_threading argument to point to correct paths of the headers and the library.

Context:

• Related to Add support nogil mode in JAX for Python 3.13 jax-ml/jax#23073 and building jaxlib with python 3.13 interpreter built with disabled GIL.

cc @vam-google

[aignas]

We also have flag_values in the python/versions.bzl used for each platform. Does this mean that we should be able to add 2 toolchains for linux, where one would be without GIL? Is no-GIL supposed to be available for all OSes?

I personally think that it would be nice to figure out how the toolchain side of things is configured as well what this PR contains.

[vfdev-5]

@aignas sorry for late reply. Maybe, this part of docs can answer your questions: https://docs.python.org/3.13/whatsnew/3.13.html#free-threaded-cpython

There are two python interpreters: default with GIL and free-threading (or no-gil) for linux, windows, macosx.

I'm not very familiar with flag_values in the python/versions.bzl, so if you think we should use it instead of something from this PR, I'm happy to implement that.

Concerning the toolchain side, I compiled jax with python 3.13t specifying manually the toolchain as following (WORKSPACE):

...

load("@xla//third_party/py:python_repo.bzl", "custom_python_interpreter")
custom_python_interpreter(
    name = "python_dev",
    urls = ["https://www.python.org/ftp/python/{version}/Python-{version}{version_variant}.tgz"],
    strip_prefix = "Python-{version}{version_variant}",
    version = "3.13.0",
    version_variant = "rc1",
    configure_params = ["--enable-optimizations", "--disable-gil"],
)

load("@xla//third_party/py:python_init_toolchains.bzl", "python_init_toolchains")
python_init_toolchains()

load("@rules_python//python:repositories.bzl", "python_register_toolchains")
python_register_toolchains(
    name = "python",
    # By default assume the interpreter is on the local file system, replace
    # with proper URL if it is not the case.
    base_url = "file://",
    ignore_root_user_error = True,
    python_version = "3.13.0",
    tool_versions = {
        "3.13.0": {
            # Path to .tar.gz with Python binary. By default it points to .tgz
            # file in cache where it was built originally; replace with proper
            # file location, if you moved it somewhere else.
            "url": "/root/.cache/bazel/_bazel_root/bee4ad1fd43279be7a03b33426e824d5/external/python_dev/python_dev-3.13.0.tgz",
            "sha256": {
                # By default we assume Linux x86_64 architecture, eplace with
                # proper architecture if you were building on a different platform.
                "x86_64-unknown-linux-gnu": "87a8ea943b6968bf07f0f483273c67fb285fa5a6d47e94228611c4dd23e65310",
            },
            "strip_prefix": "python_dev-3.13.0",
        },
    },
    free_threading = True,
)
...

[aignas]

I see that there is a new ABI tag for wheels built for the free-threaded interpreter that is called cp313t. This means that we may want to distinguish the said wheels in the PyPI integration code. However, that probably can be done in a separate PR.

I do think that the current implementation is good looking from the outside, but I am concerned that we cannot have 3.13.0 with gil and without registered at the same time. That suggests that we don't have a good way to model ABI flavours within our toolchain code.

The flag_values would have to land here:
https://github.com/bazelbuild/rules_python/blob/main/python/private/py_toolchain_suite.bzl#L35

However, the only contributing member is PLATFORMS:
https://github.com/bazelbuild/rules_python/blob/main/python/private/toolchains_repo.bzl#L74

We should probably add a config setting which would be something like //python/config_settings:gil and have 2 values - yes, no and default to yes. For now we would have to modify each value like https://github.com/bazelbuild/rules_python/blob/main/python/versions.bzl#L517
to have a yes as a flag_value in the platform and then have a way to manipulate the PLATFORMS so that a user can specify extra platforms as an input to python_register_toolchains.

We should also add it to the bzlmod python extension, but that has to wait for #2151.

So whilst this could be a good PR as a stop gap solution, the quality bar needs to be raised for long term inclusion to rules_python.

[vfdev-5]

@aignas Thanks for detailed reply!

and then have a way to manipulate the PLATFORMS so that a user can specify extra platforms as an input to python_register_toolchains.

What would be the best API for that, can you please provide an example?

python_register_toolchains(
    name = "python_3_13",
    python_version = "3.13.0",
)

python_register_toolchains(
    name = "python_3_13_ft",
    python_version = "3.13.0",
    tool_versions = {
        "3.13.0": {
            flag_values: {Label("//python/config_settings:gil"): "no",}
         }
   }
)

?

[aignas]

Sorry it's been a while. We have landed a few PRs and I think we could now extend the API in

rules_python/examples/bzlmod/MODULE.bazel

Line 61 in 18380a2

python.single_version_override(

to be something like:

python.single_version_override(
    python_version = "3.13.0",
    sha256 = {
        "aarch64-apple-darwin": "deadbeef",
        "aarch64-unknown-linux-gnu": "deadb00f",
    },
    flag_values = {
        "@rules_python//python/config_settings:gil": "no",
    },  # Added when registering the toolchains
    suffix = "_nogil", # Added as the suffix for python versions
    urls = ["https://example.org/cpython-th-{python_version}+20220227-{platform}-{build}.tar.gz"],
)

The WORKSPACE implementation would modify the TOOL_VERSIONS structure to also allow specifying the flag_values as part of a single version definition. That means that we could in theory set a value of python_repository to tell it to append t to the places which you have identified.

Previously I thought we would have to use PLATFORMS in some way, but thinking about it for the second time, I am not sure that was correct.

[vfdev-5]

@aignas I'm trying to add free-threading support to build xla and jax, they are using WORKSPACE and python_register_toolchains, right now there is no way to download free-threading version of cpython for 3.13, there is no entry in TOOL_VERSIONS corresponding to what we want (maybe the next release of indygreg will include it: indygreg/python-build-standalone#326). So, I have to build python interpreter using custom_python_interpreter from xla. Finally, I use python_register_toolchains as following:

load("@rules_python//python:repositories.bzl", "python_register_toolchains")
python_register_toolchains(
    name = "python",
    # By default assume the interpreter is on the local file system, replace
    # with proper URL if it is not the case.
    base_url = "file://",
    ignore_root_user_error = True,
    python_version = "3.13.0",
    tool_versions = {
        "3.13.0": {
            # Path to .tar.gz with Python binary. By default it points to .tgz
            # file in cache where it was built originally; replace with proper
            # file location, if you moved it somewhere else.
            "url": "/root/.cache/bazel/_bazel_root/83d0ab0b1a11dcb5adba61b76e88291b/external/python_dev/python_dev-3.13.0.tgz",
            "sha256": {
                # By default we assume Linux x86_64 architecture, eplace with
                # proper architecture if you were building on a different platform.
                "x86_64-unknown-linux-gnu": "01d224baa3ff5ddb3fc1489d8e07cbbaf17cc705652dfe0369663a658db26d49",
            },
            "strip_prefix": "python_dev-3.13.0",
        },
    },
)

to specify wanted python.
How and where can we add the flag values in this case? Appologies for silly question, I'm not fully aware of the design and the codebase of rules_python. Thanks!

EDIT:
Few more questions:

• What is the purpose of suffix argument (used in your example as suffix = "_nogil") ?
• I assume that "@rules_python//python/config_settings:gil": "no", is responsible of adding t postfix to
  • distutils_path = "lib/python{}t/distutils/distutils.cfg"
  • define_hermetic_runtime_toolchain_impl

[vfdev-5]

@aignas I tried to implement flag_values in toolchaines.
Here is the how it is invoked from XLA now:

python_register_toolchains(
    name = "python",
    # By default assume the interpreter is on the local file system, replace
    # with proper URL if it is not the case.
    base_url = "file://",
    ignore_root_user_error = True,
    python_version = "3.13.0",
    tool_versions = {
        "3.13.0": {
            # Path to .tar.gz with Python binary. By default it points to .tgz
            # file in cache where it was built originally; replace with proper
            # file location, if you moved it somewhere else.
            "url": "/root/.cache/bazel/_bazel_root/83d0ab0b1a11dcb5adba61b76e88291b/external/python_dev/python_dev-3.13.0.tgz",
            "sha256": {
                # By default we assume Linux x86_64 architecture, eplace with
                # proper architecture if you were building on a different platform.
                "x86_64-unknown-linux-gnu": "01d224baa3ff5ddb3fc1489d8e07cbbaf17cc705652dfe0369663a658db26d49",
            },
            "strip_prefix": "python_dev-3.13.0",
            # Added when registering the toolchains
            "flag_values": {
                "@rules_python//python/config_settings:free_threading": "yes",
            },
            # Added as the suffix for python versions
            "suffix": "_ft",
        },
    },
)

Can you please review and leave a feedback whether it corresponds to your idea?
I have also few questions here: #2129 (comment)
if you can answer it would be helpful!
Thanks