Skip to content

Commit

Permalink
feat(pip): support specifying requirements files per (os, arch)
Browse files Browse the repository at this point in the history
This change implements the necessary selection of the requirement files
based on the (os, arch) for both - bzlmod and legacy workspace
code paths. As part of this addition I have moved some tests from
integration tests to unit tests as now all of the logic on selecting the
right requirement file, ensuring that there are no duplicate requirement
lines and that the user knows what to provide is done in a single place.

This should be a non-breaking change for most users unless they have
been passing `requirements_linux` together with `extra_pip_args =
["--platform=manylinux_2_4_x86_64"]`, in which case they would have to
change their code to use `requirements_lock` attribute which itself is a
trivial change.
  • Loading branch information
aignas committed May 19, 2024
1 parent ede1163 commit 6e31814
Show file tree
Hide file tree
Showing 27 changed files with 1,071 additions and 382 deletions.
9 changes: 7 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,12 @@ A brief description of the categories of changes:
* (toolchains) Optional toolchain dependency: `py_binary`, `py_test`, and
`py_library` now depend on the `//python:exec_tools_toolchain_type` for build
tools.

* (deps): Bumped `bazel_skylib` to 1.6.1.
* (bzlmod): The `python` and internal `rules_python` extensions have been
marked as `reproducible` and will not include any lock file entries from now
on.

### Fixed

* (gazelle) Remove `visibility` from `NonEmptyAttr`.
Now empty(have no `deps/main/srcs/imports` attr) `py_library/test/binary` rules will
be automatically deleted correctly. For example, if `python_generation_mode`
Expand Down Expand Up @@ -64,9 +62,16 @@ A brief description of the categories of changes:
`transitive_pyc_files`, which tell the pyc files a target makes available
directly and transitively, respectively.
* `//python:features.bzl` added to allow easy feature-detection in the future.
* (pip) Allow specifying the requirements by (os, arch) and add extra
validations when parsing the inputs. This is a non-breaking change for most
users unless they have been passing multiple `requirements_*` files together
with `extra_pip_args = ["--platform=manylinux_2_4_x86_64"]`, that was an
invalid usage previously but we were not failing the build. From now on this
is explicitly disallowed.

[precompile-docs]: /precompiling


## [0.32.2] - 2024-05-14

[0.32.2]: https://github.com/bazelbuild/rules_python/releases/tag/0.32.2
Expand Down
8 changes: 5 additions & 3 deletions MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,11 @@ pip.parse(
experimental_index_url = "https://pypi.org/simple",
hub_name = "rules_python_publish_deps",
python_version = "3.11",
requirements_darwin = "//tools/publish:requirements_darwin.txt",
requirements_lock = "//tools/publish:requirements.txt",
requirements_windows = "//tools/publish:requirements_windows.txt",
requirements_by_platform = {
"//tools/publish:requirements.txt": "linux_*",
"//tools/publish:requirements_darwin.txt": "osx_*",
"//tools/publish:requirements_windows.txt": "windows_*",
},
)
use_repo(pip, "rules_python_publish_deps")

Expand Down
74 changes: 55 additions & 19 deletions docs/sphinx/pip.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,41 @@ load("@pip_deps//:requirements.bzl", "install_deps")
install_deps()
```

For `bzlmod` an equivalent `MODULE.bazel` would look like:
```starlark
pip = use_extension("//python/extensions:pip.bzl", "pip")
pip.parse(
hub_name = "pip_deps",
requirements_lock = ":requirements.txt",
)
use_repo(pip, "pip_deps")
```

You can then reference installed dependencies from a `BUILD` file with:

```starlark
load("@pip_deps//:requirements.bzl", "requirement")

py_library(
name = "bar",
...
deps = [
"//my/other:dep",
"@pip_deps//requests",
"@pip_deps//numpy",
],
)
```

The rules also provide a convenience macro for translating the entries in the
`requirements.txt` file (e.g. `opencv-python`) to the right bazel label (e.g.
`@pip_deps//opencv_python`). The convention of bazel labels is lowercase
`snake_case`, but you can use the helper to avoid depending on this convention
as follows:

```starlark
load("@pip_deps//:requirements.bzl", "requirement")

py_library(
name = "bar",
...
Expand All @@ -35,33 +65,39 @@ py_library(
)
```

In addition to the `requirement` macro, which is used to access the generated `py_library`
target generated from a package's wheel, The generated `requirements.bzl` file contains
functionality for exposing [entry points][whl_ep] as `py_binary` targets as well.
If you would like to access [entry points][whl_ep], see the `py_console_script_binary` rule documentation.

[whl_ep]: https://packaging.python.org/specifications/entry-points/

(per-os-arch-requirements)=
## Requirements for a specific OS/Architecture

In some cases you may need to use different requirements files for different OS, Arch combinations. This is enabled via the `requirements_by_platform` attribute in `pip.parse` extension and the `pip_parse` repository rule. The keys of the dictionary are labels to the file and the values are a list of comma separated target (os, arch) tuples.

For example:
```starlark
load("@pip_deps//:requirements.bzl", "entry_point")

alias(
name = "pip-compile",
actual = entry_point(
pkg = "pip-tools",
script = "pip-compile",
),
)
# ...
requirements_by_platform = {
"requirements_linux_x86_64.txt": "linux_x86_64",
"requirements_osx.txt": "osx_*",
"requirements_linux_exotic.txt": "linux_exotic",
"requirements_some_platforms.txt": "linux_aarch64,windows_*",
},
# For the list of standard platforms that the rules_python has toolchains for, default to
# the following requirements file.
requirements_lock = "requirements_lock.txt",
```

Note that for packages whose name and script are the same, only the name of the package
is needed when calling the `entry_point` macro.
In case of duplicate platforms, `rules_python` will raise an error as there has
to be unambiguous mapping of the requirement files to the (os, arch) tuples.

An alternative way is to use per-OS requirement attributes.
```starlark
load("@pip_deps//:requirements.bzl", "entry_point")

alias(
name = "flake8",
actual = entry_point("flake8"),
# ...
requirements_windows = "requirements_windows.txt",
requirements_darwin = "requirements_darwin.txt",
# For the remaining platforms (which is basically only linux OS), use this file.
requirements_lock = "requirements_lock.txt",
)
```

Expand Down
24 changes: 11 additions & 13 deletions examples/bzlmod/MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,17 @@ pip.parse(
],
hub_name = "pip",
python_version = "3.9",
requirements_lock = "//:requirements_lock_3_9.txt",
requirements_windows = "//:requirements_windows_3_9.txt",
# The requirements files for each platform that we want to support.
requirements_by_platform = {
# Default requirements file for needs to explicitly provide the platforms
"//:requirements_lock_3_9.txt": "linux_*,osx_*",
# This API allows one to specify additional platforms that the users
# configure the toolchains for themselves. In this example we add
# `windows_aarch64` to illustrate that `rules_python` won't fail to
# process the value, but it does not mean that this example will work
# on Windows ARM.
"//:requirements_windows_3_9.txt": "windows_x86_64,windows_aarch64",
},
# These modifications were created above and we
# are providing pip.parse with the label of the mod
# and the name of the wheel.
Expand Down Expand Up @@ -193,14 +202,3 @@ local_path_override(
module_name = "other_module",
path = "other_module",
)

# =====
# Config for testing duplicate packages in requirements
# =====
#
pip.parse(
hub_name = "dupe_requirements",
python_version = "3.9", # Must match whatever is marked is_default=True
requirements_lock = "//tests/dupe_requirements:requirements.txt",
)
use_repo(pip, "dupe_requirements")
19 changes: 0 additions & 19 deletions examples/bzlmod/tests/dupe_requirements/BUILD.bazel

This file was deleted.

This file was deleted.

2 changes: 0 additions & 2 deletions examples/bzlmod/tests/dupe_requirements/requirements.in

This file was deleted.

97 changes: 0 additions & 97 deletions examples/bzlmod/tests/dupe_requirements/requirements.txt

This file was deleted.

1 change: 1 addition & 0 deletions examples/pip_parse/MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ use_repo(

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
download_only = True,
experimental_requirement_cycles = {
"sphinx": [
"sphinx",
Expand Down
3 changes: 1 addition & 2 deletions examples/pip_parse_vendored/requirements.bzl
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
"""Starlark representation of locked requirements.
@generated by rules_python pip_parse repository rule
from @//:requirements.txt
@generated by rules_python pip_parse repository rule.
"""

load("@rules_python//python:pip.bzl", "pip_utils")
Expand Down
2 changes: 1 addition & 1 deletion python/pip_install/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ bzl_library(
srcs = ["pip_repository.bzl"],
deps = [
":repositories_bzl",
":requirements_parser_bzl",
"//python:repositories_bzl",
"//python:versions_bzl",
"//python/pip_install/private:generate_group_library_build_bazel_bzl",
Expand All @@ -32,6 +31,7 @@ bzl_library(
"//python/private:bzlmod_enabled_bzl",
"//python/private:envsubst_bzl",
"//python/private:normalize_name_bzl",
"//python/private:parse_requirements_bzl",
"//python/private:parse_whl_name_bzl",
"//python/private:patch_whl_bzl",
"//python/private:render_pkg_aliases_bzl",
Expand Down
Loading

0 comments on commit 6e31814

Please sign in to comment.