flare-capa is incompatible with flare-floss 3.1.0

[uckelman-sf]

Description

flare-floss 3.1.0 requires pydantic 2.6.0. No released version of flare-capa permits using pydantic 2.6.0; therefore, the two current releases cannot be used together.

Steps to Reproduce

Minimal pyproject.toml:

[tool.poetry]
name = "test"
version = "1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = ">=3.11"
flare-capa = "*"
flare-floss = ">=3"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Then:

$ poetry lock

Expected behavior:

flare-capa and flare-floss are installed.

Actual behavior:

Because no versions of flare-capa match <1.0.0 || >1.0.0,<1.2.0 || >1.2.0,<1.3.0 || >1.3.0,<1.4.0 || >1.4.0,<1.5.1 || >1.5.1,<1.6.0 || >1.6.0,<1.6.1 || >1.6.1,<1.6.2 || >1.6.2,<1.6.3 || >1.6.3,<2.0.0 || >2.0.0,<3.0.0 || >3.0.0,<3.0.1 || >3.0.1,<3.0.2 || >3.0.2,<3.0.3 || >3.0.3,<3.1.0 || >3.1.0,<3.2.0 || >3.2.0,<3.2.1 || >3.2.1,<4.0.0 || >4.0.0,<4.0.1 || >4.0.1,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1
 and flare-capa (1.0.0) depends on networkx (2.2), flare-capa (<1.2.0 || >1.2.0,<1.3.0 || >1.3.0,<1.4.0 || >1.4.0,<1.5.1 || >1.5.1,<1.6.0 || >1.6.0,<1.6.1 || >1.6.1,<1.6.2 || >1.6.2,<1.6.3 || >1.6.3,<2.0.0 || >2.0.0,<3.0.0 || >3.0.0,<3.0.1 || >3.0.1,<3.0.2 || >3.0.2,<3.0.3 || >3.0.3,<3.1.0 || >3.1.0,<3.2.0 || >3.2.0,<3.2.1 || >3.2.1,<4.0.0 || >4.0.0,<4.0.1 || >4.0.1,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2).
And because flare-capa (1.2.0) depends on networkx (2.2)
 and flare-capa (1.3.0) depends on networkx (2.2), flare-capa (<1.4.0 || >1.4.0,<1.5.1 || >1.5.1,<1.6.0 || >1.6.0,<1.6.1 || >1.6.1,<1.6.2 || >1.6.2,<1.6.3 || >1.6.3,<2.0.0 || >2.0.0,<3.0.0 || >3.0.0,<3.0.1 || >3.0.1,<3.0.2 || >3.0.2,<3.0.3 || >3.0.3,<3.1.0 || >3.1.0,<3.2.0 || >3.2.0,<3.2.1 || >3.2.1,<4.0.0 || >4.0.0,<4.0.1 || >4.0.1,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2).
And because flare-capa (1.4.0) depends on networkx (2.2)
 and flare-capa (1.5.1) depends on networkx (2.2), flare-capa (<1.6.0 || >1.6.0,<1.6.1 || >1.6.1,<1.6.2 || >1.6.2,<1.6.3 || >1.6.3,<2.0.0 || >2.0.0,<3.0.0 || >3.0.0,<3.0.1 || >3.0.1,<3.0.2 || >3.0.2,<3.0.3 || >3.0.3,<3.1.0 || >3.1.0,<3.2.0 || >3.2.0,<3.2.1 || >3.2.1,<4.0.0 || >4.0.0,<4.0.1 || >4.0.1,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2).
And because flare-capa (1.6.0) depends on networkx (2.2)
 and flare-capa (1.6.1) depends on networkx (2.2), flare-capa (<1.6.2 || >1.6.2,<1.6.3 || >1.6.3,<2.0.0 || >2.0.0,<3.0.0 || >3.0.0,<3.0.1 || >3.0.1,<3.0.2 || >3.0.2,<3.0.3 || >3.0.3,<3.1.0 || >3.1.0,<3.2.0 || >3.2.0,<3.2.1 || >3.2.1,<4.0.0 || >4.0.0,<4.0.1 || >4.0.1,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2).
And because flare-capa (1.6.2) depends on networkx (2.2)
 and flare-capa (1.6.3) depends on networkx (2.2), flare-capa (<2.0.0 || >2.0.0,<3.0.0 || >3.0.0,<3.0.1 || >3.0.1,<3.0.2 || >3.0.2,<3.0.3 || >3.0.3,<3.1.0 || >3.1.0,<3.2.0 || >3.2.0,<3.2.1 || >3.2.1,<4.0.0 || >4.0.0,<4.0.1 || >4.0.1,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2).
And because flare-capa (2.0.0) depends on pefile (2021.5.24)
 and flare-capa (3.0.0) depends on pefile (2021.9.3), flare-capa (<3.0.1 || >3.0.1,<3.0.2 || >3.0.2,<3.0.3 || >3.0.3,<3.1.0 || >3.1.0,<3.2.0 || >3.2.0,<3.2.1 || >3.2.1,<4.0.0 || >4.0.0,<4.0.1 || >4.0.1,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2) or pefile (2021.5.24 || 2021.9.3).
And because flare-capa (3.0.1) depends on pefile (2021.9.3)
 and flare-capa (3.0.2) depends on pefile (2021.9.3), flare-capa (<3.0.3 || >3.0.3,<3.1.0 || >3.1.0,<3.2.0 || >3.2.0,<3.2.1 || >3.2.1,<4.0.0 || >4.0.0,<4.0.1 || >4.0.1,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2) or pefile (2021.5.24 || 2021.9.3).
And because flare-capa (3.0.3) depends on pefile (2021.9.3)
 and flare-capa (3.1.0) depends on pefile (2021.9.3), flare-capa (<3.2.0 || >3.2.0,<3.2.1 || >3.2.1,<4.0.0 || >4.0.0,<4.0.1 || >4.0.1,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2) or pefile (2021.5.24 || 2021.9.3).
And because flare-capa (3.2.0) depends on pefile (2021.9.3)
 and flare-capa (3.2.1) depends on pefile (2021.9.3), flare-capa (<4.0.0 || >4.0.0,<4.0.1 || >4.0.1,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2) or pefile (2021.5.24 || 2021.9.3).
And because flare-capa (4.0.0) depends on pydantic (1.9.1)
 and flare-capa (4.0.1) depends on pydantic (1.9.1), flare-capa (<5.0.0 || >5.0.0,<5.1.0 || >5.1.0,<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2) or pefile (2021.5.24 || 2021.9.3) or pydantic (1.9.1).
And because flare-capa (5.0.0) depends on pydantic (1.10.4)
 and flare-capa (5.1.0) depends on pydantic (1.10.7), flare-capa (<6.0.0 || >6.0.0,<6.1.0 || >6.1.0,<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2) or pefile (2021.5.24 || 2021.9.3) or pydantic (1.9.1 || 1.10.4 || 1.10.7).
And because flare-capa (6.0.0) depends on pydantic (1.10.9)
 and flare-capa (6.1.0) depends on pydantic (2.1.1), flare-capa (<7.0.0 || >7.0.0,<7.0.1 || >7.0.1) requires networkx (2.2) or pefile (2021.5.24 || 2021.9.3) or pydantic (1.9.1 || 1.10.4 || 1.10.7 || 1.10.9 || 2.1.1).
And because flare-capa (7.0.0) depends on pydantic (2.4.0)
 and flare-capa (7.0.1) depends on pydantic (2.4.0), every version of flare-capa requires pydantic (1.9.1 || 1.10.4 || 1.10.7 || 1.10.9 || 2.1.1 || 2.4.0) or networkx (2.2) or pefile (2021.5.24 || 2021.9.3).
And because flare-floss (3.1.0) depends on both pydantic (2.6.0) and networkx (3.1), if flare-capa (*) and flare-floss (3.1.0) then pefile (2021.5.24 || 2021.9.3).
And because flare-floss (3.1.0) depends on pefile (>=2022.5.30)
 and no versions of flare-floss match >3.1, flare-capa is incompatible with flare-floss (>=3.1).
So, because test depends on both flare-capa (*) and flare-floss (>=3.1), version solving failed.

Versions

Every version of capa.

Additional Information

The immediate problem could be solved by a release of flare-capa which uses pydantic 2.6.0. However, problems of this sort could be avoided by setting constraints for dependencies which are looser.

[mr-tz]

see #2059 (comment) and #2059 (comment)

[uckelman-sf]

It looks like one could change all the == to >= in pyproject.toml, do pip install -e .[build], and then pip freeze >requirements.txt to generate a requirements.txt pinning the actual versions installed.

I'm not entirely sure yet how this interacts with building a wheel.

[williballenthin]

That sounds like a good place to start, and if the versioning is still too strict, we can relax to >=x.0.0.

👍

[uckelman-sf]

If I build a wheel with the relaxed constraints in the pyproject.toml, I can see that reflected in the flare_capa-7.0.1.dist-info/METADATA file inside the .whl. So... that's good.

[uckelman-sf]

However, the requirements.txt where I pinned exact versions isn't in the wheel at all. Given that, anybody installing the wheel won't get the dependencies from the requirements.txt. I think to get those, you'd need to do pip install -r requirements.txt after installing the wheel, which means you'd have to have requirements.txt---and you won't unless you've cloned the repo...

This is why my original suggestion was to have a separate project for the command line tool, as the pyroject.toml for that could specify exact versions and those would show up correctly in the wheel for it.

[williballenthin]

Given that, anybody installing the wheel won't get the dependencies from the requirements.txt. I think to get those, you'd need to do pip install -r requirements.txt after installing the wheel, which means you'd have to have requirements.txt

In the workflow I imagine, the user does pip install -r requirements.txt before pip install flare-capa so that when pip resolves the second time, it finds all the dependencies already present and it's basically a no-op (well installs capa but nothing else).

Agree that you'd need source code for this. And I think this is expected behavior.

I don't think users will require the pinned environment often; I imagine this only to be for the PyInstaller build and tests in CI or anyone that wants to reproduce this. Do you envision other workflows that prefer the pinned environment? I suppose active dev. What else?

[uckelman-sf]

I won't ever want the pinned environment myself. I had been assuming that you want people installing the wheel in order to use it as a command-line tool to have the the dependencies pinned. But, if you don't care about that, we've got a solution already.

I'll make a PR for that for you today or tomorrow.

[uckelman-sf]

I just pushed #2079 as a first attempt. I'm hoping that I got the pip install -r requirements.txt in the correct place for building the binary packages.

[williballenthin]

To be clear about how I'm thinking about the pinned version: we want some way to be able to assert that the environment in which all tests pass (in CI) is the same that we distribute in the standalone exe (which is the primary distribution method). We also want to make it easy for devs to recreate this environment to triage test failures or other bug reports. requirements.txt enables this because source is available for each of these flows.

I don't expect most capa library users to need the pinned environment. We think most dependencies should generally work following semver and we can bump min versions when there's a particular need.

thanks @uckelman-sf for the initial PR!