Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support freethreaded toolchains and wheels #2386

Closed
4 tasks done
aignas opened this issue Nov 8, 2024 · 0 comments · Fixed by #2460 or #2497
Closed
4 tasks done

Support freethreaded toolchains and wheels #2386

aignas opened this issue Nov 8, 2024 · 0 comments · Fixed by #2460 or #2497

Comments

@aignas
Copy link
Collaborator

aignas commented Nov 8, 2024 •
edited
Loading

Scope:

Better support for sdist building should be done separately.
@aignas aignas mentioned this issue Nov 8, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Nov 11, 2024
@aignas
feat(toolchain): support freethreaded toolchains (#2372)
2af0020 
Before this PR freethreaded toolchains were not possible to be used,
this adds the minimum plumbing to get the things working. Coverage
support is also added.

Whilst at it:
- Add plumbing to print checksums only for a particular python version.
- Bump the remaining toolchain versions that used to use the 20241008
release
- Pass around the loaded platform list so that we are only defining
toolchains for the platforms that we have loaded the hermetic toolchain
for.

Tested:
```
$ bazel run --//python/config_settings:python_version=3.13.0 --//python/config_settings:py_freethreaded="yes" //python/private:current_interpreter_executable
...
Python 3.13.0 experimental free-threading build (main, Oct 16 2024, 03:26:14) [Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
```

Closes #2129.
Work towards #2386.
@aignas aignas mentioned this issue Nov 12, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Nov 13, 2024
@aignas @rickeylev
refactor(pkg_aliases): create a macro for creating whl aliases (#2391)
7b95ea6 
This just cleans up the code and moves more logic from the
repository_rule
(i.e. generation of `BUILD.bazel` files) to loading time (macro
evaluation).
This makes the unit testing easier and I plan to also move the code that
is
generating config setting names from filenames to this new macro, but
wanted to
submit this PR to reduce the review chunks.

Summary:
- Add a new `pkg_aliases` macro.
- Move logic and tests for creating WORKSPACE aliases.
- Move logic and tests bzlmod aliases.
- Move logic and tests bzlmod aliases with groups.
- Add a test for extra alias creation.
- Use `whl_alias` in `pypi` extension integration tests.
- Improve the serialization of `whl_alias` for passing to the pypi hub
repo.

Related to #260, #2386, #2337, #2319 - hopefully cleaning the code up
will make
it easier to address those feature requests later.

---------

Co-authored-by: Richard Levasseur <richardlev@gmail.com>
@aignas aignas mentioned this issue Nov 14, 2024
Merged
aignas added a commit to aignas/rules_python that referenced this issue Nov 15, 2024
@aignas
refactor(toolchain): use bazel to extract zstd archives
ee2f98d 
Here we remove the dependence on the `zstd` archive downloaded from GH
for extracting `zstd` toolchains because bazel now supports `zstd`
extraction for a long time.

This mainly cleans up the code which is again used a lot more due to bazelbuild#2386.
@aignas aignas mentioned this issue Nov 15, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Nov 18, 2024
@aignas @rickeylev
refactor(toolchain): use bazel to extract zstd archives (#2412)
79bd1f5 
Here we remove the dependence on the `zstd` archive downloaded from GH
for extracting `zstd` toolchains because bazel now supports `zstd`
extraction for a long time.

This mainly cleans up the code which is again used a lot more due to
#2386.

Co-authored-by: Richard Levasseur <rlevasseur@google.com>
github-merge-queue bot pushed a commit that referenced this issue Nov 18, 2024
@aignas @rickeylev
refactor!(toolchain): remove uname dep in the repository_rule stage (#…
68d1b41 
…2406)

Before this PR we would shell out to `uname` on UNIX systems to get the
`arch`
of the toolchain - on Windows we would not need to do it because there
used to
be only a single Windows platform. With this change we can correctly
support
the resolution of the python interpreter on various platforms and I have
also
added an env variable to customize the selection, so that users can use
`musl`
or a `freethreaded` interpreter if they wish.

As part of this change, I have restricted visibility of the config
settings
used in the toolchain alias repo so that we are creating fewer targets.
This is
a very good time to do this before `1.0.0`.

Fixes #2145
Work towards #2276
Work towards #2386
Work towards #1211 to unblock #2402
Work towards #1361

---------

Co-authored-by: Richard Levasseur <richardlev@gmail.com>
aignas added a commit to aignas/rules_python that referenced this issue Dec 1, 2024
@aignas
feat(pypi): support freethreaded in experimental_index_url
34ea5b2 
With this we:
* Fix the previous behaviour where `abi3` wheels would be selected when
  freethreaded builds are selected. Whilst this may work in practise
  sometimes, I am not sure it has been supported by reading PEP703.
* Start selecting `cp313t` wheels when we scan what is available on PyPI.
* Ensure that the `whl_library` repository rule handles `cp313t` wheel
  extraction.
* Generate `cp313t` config_settings so that we can use them in
  `pkg_aliases`.
* Generate `cp313t` references in `pkg_aliases` macro.
* Add the 3.13 deps to dev_pip for testing.

Also tested by manually running:
```
$ bazel cquery --//python/config_settings:python_version=3.13 --//python/config_settings:py_freethreaded=yes 'kind("py_library rule", deps(@dev_pip//markupsafe))'
INFO: Analyzed target @@_main~pip~dev_pip//markupsafe:markupsafe (3 packages loaded, 4091 targets configured).
INFO: Found 1 target...
@@_main~pip~dev_pip_313_markupsafe_cp313_cp313t_manylinux_2_17_x86_64_c0ef13ea//:pkg (008c5a5)
$bazel build --//python/config_settings:python_version=3.13 --//python/config_settings:py_freethreaded=yes @dev_pip//markupsafe
```

Fixes bazelbuild#2386
@aignas aignas mentioned this issue Dec 1, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Dec 7, 2024
@aignas
feat(pypi): support freethreaded in experimental_index_url (#2460)
094a256 
With this we:
* Fix the previous behaviour where `abi3` wheels would be selected when
  freethreaded builds are selected. Whilst this may work in practise
  sometimes, I am not sure it has been supported by reading PEP703.
* Start selecting `cp313t` wheels when we scan what is available on
PyPI.
* Ensure that the `whl_library` repository rule handles `cp313t` wheel
  extraction.
* Generate `cp313t` config_settings so that we can use them in
  `pkg_aliases`.
* Generate `cp313t` references in `pkg_aliases` macro.
* Add the 3.13 deps to dev_pip for testing.

Also tested by manually running:
```
$ bazel cquery --//python/config_settings:python_version=3.13 --//python/config_settings:py_freethreaded=yes 'kind("py_library rule", deps(@dev_pip//markupsafe))'
INFO: Analyzed target @@_main~pip~dev_pip//markupsafe:markupsafe (3 packages loaded, 4091 targets configured).
INFO: Found 1 target...
@@_main~pip~dev_pip_313_markupsafe_cp313_cp313t_manylinux_2_17_x86_64_c0ef13ea//:pkg (008c5a5)
$bazel build --//python/config_settings:python_version=3.13 --//python/config_settings:py_freethreaded=yes @dev_pip//markupsafe
```

Fixes #2386
ewianda pushed a commit to ewianda/rules_python that referenced this issue Dec 7, 2024
@aignas @ewianda
feat(pypi): support freethreaded in experimental_index_url (bazelbuil…
eba7e40 
…d#2460)

With this we:
* Fix the previous behaviour where `abi3` wheels would be selected when
  freethreaded builds are selected. Whilst this may work in practise
  sometimes, I am not sure it has been supported by reading PEP703.
* Start selecting `cp313t` wheels when we scan what is available on
PyPI.
* Ensure that the `whl_library` repository rule handles `cp313t` wheel
  extraction.
* Generate `cp313t` config_settings so that we can use them in
  `pkg_aliases`.
* Generate `cp313t` references in `pkg_aliases` macro.
* Add the 3.13 deps to dev_pip for testing.

Also tested by manually running:
```
$ bazel cquery --//python/config_settings:python_version=3.13 --//python/config_settings:py_freethreaded=yes 'kind("py_library rule", deps(@dev_pip//markupsafe))'
INFO: Analyzed target @@_main~pip~dev_pip//markupsafe:markupsafe (3 packages loaded, 4091 targets configured).
INFO: Found 1 target...
@@_main~pip~dev_pip_313_markupsafe_cp313_cp313t_manylinux_2_17_x86_64_c0ef13ea//:pkg (008c5a5)
$bazel build --//python/config_settings:python_version=3.13 --//python/config_settings:py_freethreaded=yes @dev_pip//markupsafe
```

Fixes bazelbuild#2386
@aignas aignas mentioned this issue Dec 11, 2024
Merged
rickeylev pushed a commit that referenced this issue Dec 11, 2024
@aignas
doc: freethreaded support changelog (#2497)
622989e 
Support is implemented, just documenting this is left.

Fixes #2386
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@aignas