Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exp: multi-arch whl_library in bzlmod #1625

Closed
wants to merge 95 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
e97a75b
add some thoughts
aignas Dec 9, 2023
ea0f2ba
wip
aignas Dec 9, 2023
bc41145
continue to spike
aignas Dec 9, 2023
e29ae54
continue experimenting, the group libraries do not work but I think i…
aignas Dec 9, 2023
393446b
hack through visibility to get tests almost passing
aignas Dec 10, 2023
3aa23a2
document analysis of the current approach and a new approach
aignas Dec 10, 2023
1e9f6fd
start implementing a new approach
aignas Dec 10, 2023
d78f282
wip
aignas Dec 11, 2023
ff2d000
work out the visibility and cleanup
aignas Dec 11, 2023
b82ac46
wip
aignas Dec 11, 2023
75be4c7
fix visibility of the group
aignas Dec 12, 2023
c01e65a
use constants instead of strings for labels
aignas Dec 12, 2023
5b637cb
do not symlink the whl and just pass along the label
aignas Dec 12, 2023
42f916d
implement patching in minihub
aignas Dec 12, 2023
825d28d
wip
aignas Dec 14, 2023
1b91882
Merge branch 'main' into exp/whl-minihub
aignas Dec 14, 2023
0401d77
remove TODOs
aignas Dec 14, 2023
f65d6ad
wip: move the downloading of all of the wheels outside whl_library macro
aignas Dec 15, 2023
0751ad8
wip: reorder code
aignas Dec 15, 2023
764d453
add index parsing
aignas Dec 15, 2023
484a061
continue refining
aignas Dec 15, 2023
242d960
wip
aignas Dec 15, 2023
6eb77fd
Merge remote-tracking branch 'upstream/main' into exp/whl-minihub
aignas Dec 15, 2023
1c0d38b
wip: use bazel 7.1 non-blocking download to get the filenames
aignas Dec 15, 2023
89b9a79
wip: remove the os_arch_dependence on extensions
aignas Dec 15, 2023
5ba4606
wip
aignas Dec 16, 2023
0db226f
cleanup
aignas Dec 16, 2023
c1eba5e
simplify and prep to support 6.2
aignas Dec 16, 2023
2871a7c
move the parsing of all req files collectively
aignas Dec 17, 2023
aff93fa
wip
aignas Dec 17, 2023
f698068
wip
aignas Dec 17, 2023
ccf6154
Ignore the MODULE.bazel.lock
aignas Dec 17, 2023
8300db1
add notes
aignas Dec 18, 2023
f8dabc1
add docs
aignas Dec 18, 2023
d484d61
Merge branch 'main' into exp/whl-minihub
aignas Dec 18, 2023
574aff6
fixup tests
aignas Dec 18, 2023
32b8d7d
tidyup
aignas Dec 18, 2023
a54873e
make it more modular
aignas Dec 18, 2023
8e4ac60
Add a feature toggle for this feature
aignas Dec 18, 2023
c4e9524
fixup! Add a feature toggle for this feature
aignas Dec 18, 2023
3cb70b2
chore: use bazel-features for non-blocking downloading feature
aignas Dec 18, 2023
ccd5d79
cleanup downloading logic
aignas Dec 18, 2023
1bd46fb
cleanup pypi_archive code
aignas Dec 18, 2023
6ad2035
move 'get_auth' to a separate file
aignas Dec 18, 2023
5fb780d
fixup the feature flag usage
aignas Dec 18, 2023
cb91638
fixup! move 'get_auth' to a separate file
aignas Dec 18, 2023
a3d6078
Merge branch 'main' into exp/whl-minihub
aignas Dec 18, 2023
b272578
revert the bazel version to the one that we can actually support
aignas Dec 18, 2023
6ce5d11
fix(whl_library): correctly handle osx universal2 wheel platform_tag …
aignas Dec 20, 2023
2b9c686
fix(whl_library): actually apply patches if specified
aignas Dec 20, 2023
241f93a
fixup! fix(whl_library): correctly handle osx universal2 wheel platfo…
aignas Dec 20, 2023
9b24bd9
fix: demote the RECORD mismatch during patching to a warning
aignas Dec 20, 2023
ea160ce
chore: re-implement the wheel platforms in starlark
aignas Dec 20, 2023
350d006
chore: hook the platform handling in starlark
aignas Dec 20, 2023
44b995e
rename the function name
aignas Dec 20, 2023
02a00bc
remove the python implementation of target_platform
aignas Dec 20, 2023
1af04ea
chore: add the bzl_library for the new code
aignas Dec 20, 2023
2dd0791
Merge branch 'fix/universal2-mac-wheel-parsing' into exp/whl-minihub
aignas Dec 20, 2023
eda65d4
Merge branch 'fix/fix-patch-application' into exp/whl-minihub
aignas Dec 20, 2023
86747f7
fix stuff
aignas Dec 20, 2023
b19e9dd
Merge branch 'main' into exp/whl-minihub
aignas Dec 20, 2023
655e400
resolve merge conflicts and improve
aignas Dec 20, 2023
1d71aae
Use stuff form #1636
aignas Dec 20, 2023
7e60f55
add more todos and test with last_green
aignas Dec 20, 2023
28d5b9d
exp: create a symlink for the host interpreter
aignas Dec 20, 2023
a82d3d7
feat(toolchain, pip.parse): introduce a new '_host' toolchain repo
aignas Dec 21, 2023
5f65e16
updated the vendored requirements example
aignas Dec 21, 2023
0afe5b3
Merge branch 'feat/setup-a-host-interpreter-for-external-repo-usage' …
aignas Dec 21, 2023
a759cf6
remove os,arch_dep
aignas Dec 21, 2023
4daddcf
wip
aignas Dec 21, 2023
636806b
Merge branch 'main' into exp/whl-minihub
aignas Dec 21, 2023
3d6e8bd
feat(toolchain, pip.parse): introduce a new '_host' toolchain repo
aignas Dec 21, 2023
4840d42
updated the vendored requirements example
aignas Dec 21, 2023
1a97810
do the interpreter symlink resolution everywhere
aignas Dec 21, 2023
169a4c1
Merge branch 'feat/setup-a-host-interpreter-for-external-repo-usage' …
aignas Dec 21, 2023
a4af2bb
disable the print
aignas Dec 21, 2023
a923fdb
Add more text about pypi_file usage
aignas Dec 24, 2023
b7e31db
rework slightly to avoid the _label helper
aignas Dec 24, 2023
2aeda3b
more cleanup
aignas Dec 24, 2023
e778402
fixup
aignas Dec 24, 2023
12a428e
Remove a TODO note
aignas Dec 24, 2023
844913a
continue experimenting with smaller requirement lines
aignas Dec 24, 2023
f8319c3
more cleanup
aignas Dec 25, 2023
79cf3a5
wip
aignas Dec 25, 2023
dd201b0
Add a trailing slash when fetching metadata
aignas Dec 29, 2023
7392dcd
Merge branch 'main' into exp/whl-minihub
aignas Jan 9, 2024
333f35f
ignore MODULE lock files
aignas Jan 10, 2024
abf60c1
revert some of the accidental changes
aignas Jan 10, 2024
5fa7077
rebase conflicts
aignas Jan 10, 2024
6fad811
Merge branch 'main' into exp/whl-minihub
aignas Jan 10, 2024
9dca6aa
more conflicts
aignas Jan 10, 2024
91a73b8
Add a note
aignas Jan 11, 2024
e32cb68
Merge branch 'main' into exp/whl-minihub
aignas Jan 22, 2024
d66b27b
use workspace_name instead of repo_name for Label
aignas Jan 22, 2024
51474e4
Merge branch 'main' into exp/whl-minihub
aignas Jan 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
revert some of the accidental changes
aignas committed Jan 10, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit abf60c110cac3c324163e1b515faa824ac1738b1
2 changes: 1 addition & 1 deletion python/private/bzlmod/pythons_hub.bzl
Original file line number Diff line number Diff line change
@@ -99,7 +99,7 @@ def _hub_repo_impl(rctx):
(os, arch) = get_host_os_arch(rctx)
platform = get_host_platform(os, arch)
is_windows = (os == WINDOWS_NAME)
path = "python.exe" if is_windows else "python"
path = "python.exe" if is_windows else "bin/python3"

# Create a dict that is later used to create
# a symlink to a interpreter.
14 changes: 2 additions & 12 deletions python/private/toolchains_repo.bzl
Original file line number Diff line number Diff line change
@@ -157,7 +157,7 @@ def _toolchain_aliases_impl(rctx):
package(default_visibility = ["//visibility:public"])
load("@rules_python//python:versions.bzl", "gen_python_config_settings")
gen_python_config_settings()
exports_files(["defs.bzl", "host_python"])
exports_files(["defs.bzl"])

PLATFORMS = [
{loaded_platforms}
@@ -168,7 +168,7 @@ alias(name = "libpython", actual = select({{":" + item: "@{py_repository}_
alias(name = "py3_runtime", actual = select({{":" + item: "@{py_repository}_" + item + "//:py3_runtime" for item in PLATFORMS}}))
alias(name = "python_headers", actual = select({{":" + item: "@{py_repository}_" + item + "//:python_headers" for item in PLATFORMS}}))
alias(name = "python_runtimes", actual = select({{":" + item: "@{py_repository}_" + item + "//:python_runtimes" for item in PLATFORMS}}))
alias(name = "python3", actual = select({{":" + item: "@{py_repository}_" + item + "//:python" for item in PLATFORMS}}))
alias(name = "python3", actual = select({{":" + item: "@{py_repository}_" + item + "//:" + ("python.exe" if "windows" in item else "bin/python3") for item in PLATFORMS}}))
""".format(
py_repository = rctx.attr.user_repository_name,
loaded_platforms = "\n".join([" \"{}\",".format(p) for p in rctx.attr.platforms]),
@@ -180,17 +180,7 @@ alias(name = "pip", actual = select({{":" + item: "@{py_repository}_
py_repository = rctx.attr.user_repository_name,
host_platform = host_platform,
)

rctx.file("BUILD.bazel", build_contents)
host_python = rctx.path(
Label(
"@@{py_repository}_{host_platform}//:python".format(
py_repository = rctx.attr.name,
host_platform = host_platform,
),
),
)
rctx.symlink(host_python, "host_python")

# Expose a Starlark file so rules can know what host platform we used and where to find an interpreter
# when using repository_ctx.path, which doesn't understand aliases.
1 change: 0 additions & 1 deletion tests/pip_repository_entry_points/bazel-bin

This file was deleted.

1 change: 0 additions & 1 deletion tests/pip_repository_entry_points/bazel-out

This file was deleted.

This file was deleted.

1 change: 0 additions & 1 deletion tests/pip_repository_entry_points/bazel-testlogs

This file was deleted.

3 changes: 0 additions & 3 deletions tests/private/whl_target_platform/BUILD.bazel

This file was deleted.

54 changes: 0 additions & 54 deletions tests/private/whl_target_platform/whl_target_platform_tests.bzl

This file was deleted.