Add option to use "pip download" instead of "pip wheel" to download wheels for other platforms

[jesseschalken]

PR Checklist

Please check if your PR fulfills the following requirements:

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• Bugfix
• Feature (please, look at the "Scope of the project" section in the README.md file)
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• Build related changes
• CI related changes
• Documentation content changes
• Other... Please describe:

What is the current behavior?

Running pip_parse or whl_library on Linux with a requirements.txt for Windows downloads the Linux instead of the Windows version of wheels.

Adding extra_pip_args = ["--platform", "win_amd64"] doesn't work because --platform isn't a valid flag for pip wheel.

In the case that all pip dependencies are available as precompiled wheels, pip download can be used instead, which accepts the --platform flag.

Issue Number: N/A

What is the new behavior?

Add a new option download_only to pip_parse and whl_library to allow using pip download instead of pip wheel so that wheels for platforms other than the host platform can be downloaded by adding --platform ... to extra_pip_args.

Does this PR introduce a breaking change?

• Yes
• No

Other information

[UebelAndre]

@groodt would you be able to take a look here some time this week? This change would be very helpful!

(also thanks for all the support!)

[jesseschalken]

It might be useful to document how this can be used to target multiple platforms. Example:

# WORKSPACE

load("@rules_python//python:pip.bzl", "pip_parse")
load("@python3_10//:defs.bzl", "interpreter")

# Repeat for each supported target platform
pip_parse(
    name = "pip_windows",
    python_interpreter_target = interpreter,
    requirements_lock = "requirements_windows.txt",
    download_only = True,
    extra_pip_args = ["--platform", "win_amd64"],
)

# BUILD

load("@rules_python//python:pip.bzl", "compile_pip_requirements")

# Repeat for each supported target platform
compile_pip_requirements(
    name = "requirements_windows",
    requirements_in = "requirements.in",
    requirements_txt = "requirements_windows.txt",
    target_compatible_with = [
        "@platforms//os:windows",
        "@platforms//cpu:x86_64",
    ],
)

# Repeat for each supported target platform
config_setting(
    name = "windows_x86_64",
    constraint_values = [
        "@platforms//os:windows",
        "@platforms//cpu:x86_64",
    ],
)

# Repeat for each pip dependency
alias(
    name = "numpy",
    actual = select({
        # Repeat for each target platform
        ":windows_x86_64": "@pip_windows_numpy//:pkg",
        # ...
    }),
    visibility = ["//visibility:public"],
)

# ...

Obviously can be simplified with macros and loops when there are multiple target platforms and pip dependencies.

[schultetwin]

I’m a big fan of this change, or something similar! Out of curiosity, why this PR vs. #510?

This is definitely simpler! But #510 comes with the aliasing for platforms already?

[jesseschalken]

@schultetwin I didn't know about #510 and I don't have a strong preference between either PR.

It would be nice if there was a properly supported cross-build workflow like #510 is aiming for but it has been open for a year and there appears to be some open design questions.

This PR is a simpler change to at least unblock the ability to do cross builds as detailed above in #773 (comment).

[schultetwin]

This PR is a simpler change to at least unblock the ability to do cross builds as detailed above in #773 (comment).

Understood, thank you!

[groodt]

LGTM

This PR is simple and minimally invasive compared to PR #510. It unblocks users who require cross-platform builds and who are able to stand-up their own pypi mirrors / devpi pre-populated with wheels for all platforms. This is something that we probably should add documentation around and recommend.

Just for the record, both this PR and #510 still have short-comings and the maintainers of these rules are thinking hard about moving away from so much reliance on repository rules.

[sasha-wing]

Thank you for this change, it saved me

[kpark-hrp]

@jesseschalken @groodt

Is there a way to make this work with Gazelle? It would be awesome if gazelle can understand the select({...}) to work with multiple repos.