Dependency resolution when having multiple pip_parse and multi_pip_parse in the same WORKSPACE

[harshalparekh6]

🐞 bug report

Affected Rule

py_binary

Is this a regression?

N/A

Description

A clear and concise description of the problem...

We have multiple pip_parse in the WORKSPACE having two different versions of protobuf (3.x and 4.x). We rely on dependency sequencing and order the dependencies based on which version we want in the py_binary. This works fine when there are two pip_parse.

We are in the process of upgrading the python version, and moved to multi_pip_parse for one of the pip_parse. This broke the logic of dependency sequencing, and the higher version is preferred regardless of the order.

When we move both the pip_parse to multi_pip_parse, the order and sequencing logic is requested.

🔬 Minimal Reproduction

WORKSPACE

multi_pip_parse(
    name = "multi_pypi",
    default_version = default_python_version,
    python_interpreter_target = {
        "3.9": "@python_3_9_host//:python",
        "3.11": "@python_3_11_host//:python",
    },
    requirements_lock = {
        "3.9": "//build_tools/python_requirements/multi_pypi:requirements-lock.txt",
        "3.11": "//build_tools/python_requirements/multi_pypi:requirements-lock_3_11.txt",
    },
)

wrapper_pip_parse(
    name = "pypi",
    requirements_lock = "//build_tools/python_requirements/pypi:requirements-lock.txt",
)

multi_pypi:requirements-lock.txt

protobuf==3.19.6

pypi:requirements-lock.txt

protobuf==4.25.8

py_binary resolving protobuf 4.25.8, but 3.19.6 is ordered above
(if both the requirements were to come from pip_parse, this py_binary would resolve to protobuf 3.19.6

load("@multi_pypi//:requirements.bzl", multi_pypi_requirement = "requirement") # protobuf==3.19.6
load("@pypi//:requirements.bzl", pypi_requirement = "requirement") # protobuf==4.25.8
py_binary(
    name = "binary",
    ...
    deps = [
        multi_pypi_requirement("protobuf"),
        pypi_requirement("protobuf"),
    ],
    ...
)

🔥 Exception or Error

N/A


## 🌍  Your Environment

**Operating System:**
Linux

**Output of `bazel version`:**
7.5.4

**Rules python version:**
1.5.4

  

Rules_python version:

Anything else relevant?

[rickeylev]

We rely on dependency sequencing and order the dependencies based on which version we want in the py_binary.

The order isn't defined. Under the hood, the default depset order is used.

depending on two different pip_parse repos

Well, this is the crux of your problem. This is going to add duplicate versions of libraries. You can only have one closure of dependencies; if there's multiple its undefined which will be used (plus it'll bloat your binary size). The best option today is to use transitions and select() to ensure a single consistent closure is used.

[harshalparekh6]

@rickeylev I was going by this comment:

# Remove duplicates to avoid overly long PYTHONPATH (#10977). Preserve order,
  # keep first occurrence only.

The best option today is to use transitions and select() to ensure a single consistent closure is used.

Can you point to some example in the docs or have a snippet handy?

[rickeylev]

That code snippet in the bootstrap preserves the order as passed to it by bazel, but the order from Bazel isn't well defined.

example

You'd want to do something like this:

https://rules-python.readthedocs.io/en/latest/howto/common-deps-with-multiple-pypi-versions.html

That examples assumes rules_python 1.6 and bzlmod, which provide some conveniences for making it easier and more transparent (the docs forget to mention the arg in py_repositories you can use). The same can be accomplished without those features, though. https://github.com/fmeum/with_cfg.bzl can be used to make applying a transition for a custom flag easier; the rest of the example should translate mostly as-is.

HTH