Multiple versions of same build_requires package in lockfile?

[MTomBosch]

Hello,

we are facing the issue that we are generatng a lockfile and in this lockfile we have seen that the same build_requires package is listed twice (with different versions).

After analysing this we have found out that one version is provided by the profile and one by the recipe itself.

Is that supposed to be allowed? I would have expected this to be forbidden.
Also when installing using the lockfile which version is then installed?

BTW. We are using Conan 1.59.

[memsharded]

Hi @MTomBosch

In Conan 1.X, the lockfile is a graph, and it represents the nodes of the graph. Is the build_require listed as a dependency for the same node in the graph, or it just appears twice in the graph, but used by different packages?

[MTomBosch]

We have one direct dependency coming from the recipe of a package and one global sort-of dependency coming from the profile.
But we have more information right now. One of these two is in the build context and one is from the host context.

Of course we are now checking if this is at all acceptable but independent of that it would be good to know what the expected behaviour of Conan is in such a constellation.

[memsharded]

It is totally possible for Conan to have a dependency in the build context for a version, and using a different version in the host context. They are basically independent things, the "build" context provides binaries and the "host" context provides libraries. For the cases that they must keep in sync, the recommendation would be to make sure that they use the same version in the recipe, to avoid the downstream overrides and use version-ranges as a standard approach to upgrade the recipes to the newer released versions of the dependencies.

[MTomBosch]

Ok fine, that makes totally sense. But what does this mean when I do a conan install? Is Conan then installing both versions?

I think (by looking at our logs) that only one version (the one from the recipe) is then downloaded.
That means when you say "downstream override" you mean that the version of the package defined in the recipe overrides the version in the profile although they are defined for different context? And then only the version from the recipe is then downloaded and provided in the context that the recipe has defined?

[MTomBosch]

Here is some more input after further investigation:

Conclusion: Conan (1.x) does not install all packages from a lockfile when the same package is contained twice but in two different contexts (build and host) in a lockfile.

Steps to reproduce:

Hint: Tested with conan versions 1.47.0 and 1.59.0.

Prerequisite: You have a package available in your environment (cache/remote) in two different versions for which one version can be installed with the default profile and one with settings listed the "profiles.txt" below. Create these packages in advance or exchange the name "my-pkg" with existing packages in your environment (cache/remote) matching the above criteria.
Also replace the remote "my-remote" by the valid remote in your environment or remove it when your have everyrhing in your local cache.

Create a file "profile.txt" with content:

[settings]
os=Linux
arch=x86_64

[build_requires]
some-pkg/[^3.0.0]@my-user/mychannel

Create a file "conanfile.py" with the following content:
 

from conans import ConanFile, CMake

class MyPkg(ConanFile):
    name = "my-pgk"
    description = "description"
    license = "proprietary"
    url = "https://nowhere"
    revision = "auto"
    author = "author"
    settings = "os", "arch"

    def build_requirements(self):
        self.build_requires('some-pkg/[^4.0.0]@my-user/mychannel', force_host_context=True)

Then run these commands:

1. conan install -pr:b default -pr:h ./profile.txt -u -r my-remote conanfile.py
2. grep some-pkg conan.lock
3. conan install --lockfile=conan.lock conanfile.py

Observed behaviour:

In the first conan install (without lockfile), package some-pkg is installed 2 times (v3 and v4) at the same time.
The grep shows the lockfile contains both entires.
In the second conan install (with lockfile), package some-pkg is only installed one time (only v3, v4 is missing).

Expectation:
In the second conan install (with lockfile), package some-pkg is also installed 2 times (v3 and v4) at the same time.

[memsharded]

Managed to reproduce in 1.X with:

def test_duplicated_build_host_require():
    c = TestClient()
    c.save({"tool/conanfile.py": GenConanfile("tool"),
            "pkg/conanfile.py": GenConanfile("pkg", "0.1").
                                with_build_requirement("tool/[^4.0]", force_host_context=True),
            "profile": "[tool_requires]\ntool/[^3.0]"})
    c.run("create tool 3.0@")
    c.run("create tool 4.0@")
    c.run("install pkg -pr:b=default -pr:h=profile")
    assert "tool/4.0 from local cache - Cache" in c.out
    c.run("install pkg --lockfile=conan.lock")
    assert "tool/4.0 from local cache - Cache" in c.out

Same behavior is working in 2.0, equivalent test works:

def test_duplicated_build_host_require():
    c = TestClient()
    c.save({"tool/conanfile.py": GenConanfile("tool"),
            "pkg/conanfile.py": GenConanfile("pkg", "0.1").with_test_requires("tool/[^4.0]"),
            "profile": "[tool_requires]\ntool/[^3.0]"})
    c.run("create tool --version=3.0")
    c.run("create tool --version=4.0")
    c.run("install pkg -pr:b=default -pr:h=profile --lockfile-out=conan.lock")
    c.assert_listed_require({"tool/3.0": "Cache"}, build=True)
    c.assert_listed_require({"tool/4.0": "Cache"}, test=True)
    c.run("install pkg -pr:b=default -pr:h=profile --lockfile=conan.lock")
    c.assert_listed_require({"tool/3.0": "Cache"}, build=True)
    c.assert_listed_require({"tool/4.0": "Cache"}, test=True)

[MTomBosch]

Ok. That is good news that you could reproduce it.
Can you tell me if you will fix this for 1.x so that both package versions will be installed?

[memsharded]

Can you tell me if you will fix this for 1.x so that both package versions will be installed?

We will try, but can't be guaranteed. There are other issues with lockfiles+tool_requires in 1.X that we haven't managed to fix, as the complexity of lockfiles in 1.X is huge, and the risk of breaking is too much.

[SzBosch]

@memsharded: I already found the bug in the source code and fixed it locally. I would like to provide the fix (which is very small actually) to that you can review it and include it asap into a (bugfix) release of conan 1.x.

[memsharded]

Thanks @SzBosch that is great!

[memsharded]

#13788 merged, to be released next 1.60