Allow pip_repository (and pip_parse) to "select" requirements based on both OS and architecture

[jiawen]

🚀 feature request

Relevant Rules

pip_repository and by extension pip_parse.

Description

Because pip_repository is a repository rule, it somewhat awkwardly uses several attributes to override the default requirements file based exclusively on a limited selection of operating systems (linux, macos, windows).

Now that ARM is a more popular architecture and available on all three operating systems, we need a way to select among that axis as well. Otherwise, some wheels that have native dependencies may not be available on certain architectures. We may want to consider an "os version" as a third axis as well. For example, rawpy is available on MacOS but only on Intel, and possibly only MacOS 10.9 (I don't have a machine to test).

Describe the solution you'd like

Because it's awkward to add all possible combinations as new attributes, perhaps we should consider passing in a dictionary of tuples to match against fields in repository_os. The current implementation uses only .name. It's easy to add .arch to match against arch. It might be a little tricky to add "OS version" as it requires either adding a new field to repository_os (probably possible), or invoking a platform-dependent shell script.

Describe alternatives you've considered

Passing in a struct? It's more or less the same as a dictionary.

[jiawen]

@likesum FYI.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[yk]

bump. generated requirements differ between architectures, thus making version control of e.g. lock files very difficult

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[arrdem]

In order to solve this problem we settled on using a workspace rule we call python_config, which is able to use the rctx to inspect the building host and template out a config.bzl file containing values such as a computed path to a requirements_lock.txt chosen based on the building host. This allows us to check in many platform+virtualenv combination lockfiles and pick "the right one" transparently within Bazel without modification to rules_python.

Our WORKSPACE then looks something like this -

python_version = "3.8"

load("//tools/rules:external_python.bzl", "external_python")

external_python(
    name = "local_py",
    version = python_version,
)

register_toolchains("@local_py//:python_toolchain")

# Refer to this workspace rule and associated template for how we're choosing
# the Python configuration.
load("//tools/rules:python_config.bzl", "python_config")

python_config(
    name = "pycfg",
    version = python_version,
)

load("@pycfg//:config.bzl", "environment", "extra_pip_args", "requirement_clusters", "requirements_lock")

pip_parse(
    name = "pypi",
    environment = environment,
    extra_pip_args = extra_pip_args,
    python_interpreter_target = "@local_py//:python3",
    requirement_clusters = requirement_clusters,
    requirements_lock = requirements_lock,
)

load("@pypi//:requirements.bzl", pypi_install_deps = "install_deps")

pypi_install_deps()

It would be nice if there was a better solution "in the box", but per platform+venv lockfiles appears the best way to go and it would be somewhat unreasonable to teach rules_python about all the relevant combinations. Having different requirements arguments for linux/darwin/windows is already IMO awkward.

[aignas]

FYI: #1593 added whl_library ability to point to dependencies that are platform specific, now we need installing of dependencies based on the target platform, that is further away. I had a poc done for it in #1625, but I am not sure if that design is the best way to move forward.

[thomasegriffith]

Hi @aignas - thanks for #1593! My understanding of this is that we can point to dependencies that are OS-specific, but not architecture specific. For example, one can do requirements_linux = "//linux:requirements_lock.txt" and requirements_windows = "//windows:requirements_lock.txt".

Is there a way to switch between architectures? If not, should the API expose something akin to requirements_linux_x86_64 and requirements_linux_aarch64? Perhaps that's a little messy, but something that captures the same point would be nice.

(FWIW I see the title of #1625 contains "multi-arch", but I'm not following how to actually switch between different architectures on say linux.)