[question] Overriding user and channel of a dependency in 2.0

[samuel-emrys]

What is your question?

Is there a mechanism of overriding a user and channel in a dependency in conan 2? This was possible in 1.x, and was a useful feature in the following cases without modifying the conanfile:

• Switching between recipes of the same library provided by different users (i.e, pkg/x.y.z@userA/stable vs pkg/x.y.z@userB/stable vs pkg/x.y.z@_/_)
• Switching between recipes of the same package in different stability channels (i.e, pkg/x.y.z@user/stable vs pkg/x.y.z@user/experimental)

I'm aware that lockfiles have notionally replaced the --require-override syntax, however these only seem to be appropriate for overriding versions of a dependency. Attempting to use conan lock add --requires pkg/x.y.z@user/experimental when the recipe contains self.requires("pkg/x.y.z@user/stable") just continues to install pkg/x.y.z@user/stable.

If there is no mechanism for this, what's the intended replacement to manage these workflows?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @samuel-emrys

I think the previous override mechanism still works in Conan 2.0, the only difference is that using the explicit override=True or force=True is necessary. I am proposing PR #13745 that is just a test that verifies that this is still possible and works to close this question.

A different story is that this probably wouldn't be a recommended approach. User/channel shouldn't be a versioning strategy, just part of the name. If it is a different software code, it should be a different version, if it is not different code, it should have the same binary name, and its "maturity" controlled by where it is, in which server repository it is hosted, and dependency resolution can be done against those, or explicitly pinning the revision of the thing to do. But this would be a deviation of the original issue, so please check that PR and that overrides still work and let us know.

[samuel-emrys]

Thanks for the reply @memsharded. The override=True mechanism doesn't really help me in this situation. I'm looking for a mechanism to override a requirement without modifying the conanfile (I've learnt the hard way that modifying this back when crossing the "release" boundary when merging to master can get messy). I'll elaborate on my use case a bit and perhaps you can suggest an alternative.

Considering the following conanfile:

class Foo(ConanFile):
    name = "foo"
    version = "0.1.0"
    def requirements(self):
        self.requires("pkg/[>=1.0.0 <2.0.0]@user/stable")

I use git flow to manage my libraries. This has the following implications:

• Code on master is always versioned, and the contents of a given version never changes. This corresponds to a stable branch. This means that every version is entirely unambiguous - pkg/1.0.0 will only ever point to a single version, hence the stable tag.
• Code on develop is in the staging area for the next release (whether this is a major, minor or patch release). This is versioned as such, so if it's a minor release and the code on master is version 1.0.0, then this will refer to pkg/1.1.0. However, it's not necessarily the case that this will be composed of a single commit - it may be many. Every time a package is built on develop, the contents of pkg/1.1.0 will change. This is why it gets an experimental tag for its channel. We rely on recipe revisions here to refer to individual changes within that version, but pkg/1.1.0@user/experimental will always refer to the latest change.

Because a number of my packages are inter-related, co-development of these packages can be useful and so the way I was approaching this was to use:

conan create . user/testing --require-override=pkg/1.1.0@user/experimental

(in this case, the testing channel is just to make sure anything I build locally doesn't conflict with what I get from the CI) - but this allows me to turn pkg/[>=1.0.0 <2.0.0]@user/stable into pkg/1.1.0@user/experimental in my dependency tree.

I'm pretty happy with this process - I feel as though it semantically maps reasonably well to git flow, and was supported in 1.x.

To address some of your comments:

If it is a different software code, it should be a different version

I disagree with this. This assumes a new version is released for every commit (every change to the code base), if a new package is being built for every commit. This is the approach we apply on the master branch, but on develop, often a large number of commits comprise a new version release.

if it is not different code, it should have the same binary name, and its "maturity" controlled by where it is, in which server repository it is hosted, and dependency resolution can be done against those, or explicitly pinning the revision of the thing to do

Could you illustrate what a workflow like this would look like? My best guess is:

conan create . --user=user --channel=testing -r=experimental-repo --build=pkg/1.1.0

But this has a lot of ambiguity in it. If I don't have every other dependency in my local cache, then I'm not able to be explicit about which maturity level each dependency has - all of the missing ones will be pulled from experimental-repo, which doesn't give the control that would be useful.

My use case would be satisfied by bringing --require-override back, however in the spirit of embracing the conan 2.0 way of doing things I think what I'm after could be satisfied by a small expansion of the existing version override rules:

conan lock create .

which would give:

{
    "version": "0.5",
    "requires": [
        "pkg/1.0.0@user/stable"
    ],
    "build_requires": [],
    "python_requires": []
}

And then when a dependency needs to be overridden:

conan lock add --require-override pkg/1.1.0@user/experimental

which would modify this as follows:

{
    "version": "0.5",
    "requires": [
        "pkg/1.1.0@user/experimental"
        "pkg/1.0.0@user/stable"
    ],
    "build_requires": [],
    "python_requires": []
}

then one could build using the updated lockfile:

conan build . -l conan.lock

(in my testing where trying to insert an older version, i.e. if i used pkg/0.1.0 in the above case, it's actually inserted below which wouldn't be desirable for overriding the version, but I'll save that for a separate issue)

[memsharded]

Hi @samuel-emrys

Thanks for the detailed explanation, I think I understand the case a bit better.

I'd also say that Conan 2.0 is also promoting more standard versioning strategies, in the case that you are describing, it could be like:

• master branch contains packages like pkg/1.0.0@user (Conan 2.0 allows using only user, but not channel)
• develop branch packages are not prod ready, and we want to mark that somehow. In semver this is done with a pre-release, so this wouldn't be pkg/1.1.0@user/experimental, but rather a pre-release like pkg/1.1.0-dev@user
• It is possible to do multiple pre-releases like for different branches: pkg/1.1.0-alpha, and also versioned pkg/1.1.0-beta.1.2
• The pre-release already indicates this is before 1.1.0. That is 1.1.0 has not been released yet (it will be when in master branch)
• Pre-releases have advantages over channel, for example, they are ordered (beta > alpha), and they are also ordered by their sub-versions. They can be used in version ranges.
• Revisions also work inside pre-releases, no difference here.
• Conan 2.0 has added a new feature (conf core.version_ranges:resolve_prereleases) to be able to explicitly opt-in and out of version ranges resolving to pre-releases, which is a toggle that allows testing the pre-releases or obtaining the stable versions, without having to change a single line in the recipes (this is to be documented, the feature is there, the docs are WIP), and even without needing to use a lockfile.
• It is also a good practice to have pre-releases in a different server repo, but this is not strictly necessary if using the right version ranges.

Please let me know if this helps.

[samuel-emrys]

Thanks @memsharded that's a useful response. I hadn't considered using a pre-release tag to mark development versions - it's a good idea. I think it will probably solve my immediate requirement, but I'll give it some road testing to make sure it covers my use cases before closing this issue.

It would also mean that when building locally, I can't differentiate between my local builds and what I've obtained from the server, which is a useful distinction to make because the contents of our server comes only from CI builds.

The only other gripe that I have with that solution is that it's very semver specific. There are other versioning schemes which won't preserve this kind of ordering, many of which are adopted by libraries throughout the conan index. This means that we'll only be able to override packages that conform to semver using this method. It also assumes that I have control over the versioning scheme of my (or my organisations) libraries, which may not always be the case, so it's limited in those respects.

[memsharded]

There are other versioning schemes which won't preserve this kind of ordering, many of which are adopted by libraries throughout the conan index.

Conan 2.0 has made version range resolution more flexible, so it will understand things like cci.xxx.number without major issues, and will be able to order and resolve the number part correctly too. Hope this helps too.

But yes, in any case, as to create a new channel anyway it can only happen in the conan export and conan create commands, that already works over a local conanfile.py recipe from source, at the moment it is being created, so it assumes that in the same way that the channel can be defined, a variation of the version might be defined. This shouldn't be an issue for conan-center, as the version is always dynamic, but for your own recipes it might need to make sure that the version can also be dynamic, either by not defining it or by using set_version().

[memsharded]

This has been closed as a result of merging #13745, if there is any further comment when you give the above a try please re-open or create a new ticket. Thanks!

[samuel-emrys]

@memsharded I've been thinking about this a bit more. It does strike me that being able to override using user/channel would be a more general, less opinionated solution (in the sense that it doesn't force you to couple the versioning scheme with package variants). Is there a specific objection to this functionality from the conan team? Why was it removed?

Also, after having thought about it a bit more, it strikes me that this would fit reasonably well within a "virtual recipe" paradigm as described/requested in:

• [question] How to create a virtual like package of libjpeg & libjpeg-turbo? #12374
• [question] How to replace zlib with zlib-ng conan-center-index#14748
• add virtual-libjpeg recipe conan-center-index#14814

They're different, of course, but if you look at the fundamental principles, there's scope for a consistent approach:

• packages with the same name field implement the same library interface, therefore should be able to override each other
• packages that have the provides attribute implement the same library interface, therefore should be able to override each other

This would mean that the default case is that all packages essentially have

from conan import ConanFile

class MyPkg(ConanFile):
    name = "mypkg"
    version = "1.0"
    provides = "mypkg" # explicit for demonstration, but would be the default case and not explicitly specified

Which would allow overriding a package in the tree that varies on version, user or channel or any combination of these. But then also, it can contribute into a different interface (which is the current provides semantic):

from conan import ConanFile

class MyPkg(ConanFile):
    name = "mypkg"
    version = "1.0"
    provides = "jpeg" # also mypkg, can override packages matching `mypkg` or ones that also provide `"jpeg"`

Just some thoughts.

Edit: I just saw that this actually seems to already be the case? Could this be used to define the override mechanism?

If the attribute is omitted, the value of the attribute is assumed to be equal to the current package name. Thus, it’s redundant for libjpeg recipe to declare that it provides libjpeg, it’s already implicitly assumed by Conan.

[memsharded]

I think there are 2 different things in this thread

@memsharded I've been thinking about this a bit more. It does strike me that being able to override using user/channel would be a more general, less opinionated solution (in the sense that it doesn't force you to couple the versioning scheme with package variants). Is there a specific objection to this functionality from the conan team? Why was it removed?

Overrides are a very problematic case because it introduce variability points in places where it is almost impossible to recover the information when it is necessary. Let me put an example, and the challenge for lockfiles (that #13597 is trying to solve, and we have been working in previous PRs to provide support for this one, quite a lot of effort) that it defines:

• app/1.0->pkga/1.0->pkgb/1.0->pkgc/1.0
• At the same time app/1.0 introduces an override to pkgc/1.1
• A CI job checks what needs to be built and it computes it needs to build pkgc/1.0, pkgb/1.0, pkga/1.0 and app/1.0, in this order
• When it gets to pkga/1.0 or app/1.0, it will tell "Missing binary, I cannot find a binary for pkgb/1.0 (or for pkga/1.0)
• Why? Because when pkgb/1.0 and pkga/1.0 are built, they do not know that they depend on pkgc/1.1, but they build against pkgc/1.0. There is nothing in the profile, themselves, or their dependencies that tells them to build against pkg/1.1, so they resolve to pkgc/1.0. But app/1.0 needs binaries compiled against pkgc/1.1 otherwise it will have link errors, so app/1.0 fails with missing binaries for those dependencies.

a more general, less opinionated solution (in the sense that it doesn't force you to couple the versioning scheme with package variants)

The opinionated solution was previously the Conan one, that clearly deviated from the known good practices around versioning and versioning standards. At least for "maturity" level, the pre-releases are quite a standard approach across different technologies.

Also, it is important to highlight that Conan 2.0 hasn't removed the overrides, they are still very possible and the test in #13745 proves it keep working. They are possible in recipes, it has been removed from CLI args, because that makes it yet another level of un-traceability, because those values are no longer capture anywhere. They do not exist in profiles, conf, recipes, lockfiles... they only exist in the caller (CI), which makes things more complicated to track, reproduce by developers, etc.

Note that we are even discouraging overrides as a general versioning approach, check for example: https://docs.conan.io/2/tutorial/versioning/conflicts.html

Resolving version conflicts by overrides/forces should in general be the exception and avoided when possible, applied as a temporary workaround.

This is because we have saw many users during the last years struggling with using channel for this purpose, while a more standard versioning approach with pre-releases, version ranges, etc, seems to have better results in the majority of cases.

---

A completely different issue is the provides thing, when different packages (by name) can provide exactly the same interface and functionality. We have considered this problem a couple of times, but the "reference by name" is so core to the whole dependency resolution, that it would require a major complete re-write of most of the Conan core, not possible.

Edit: I just saw that this actually seems to already be the case? Could this be used to define the override mechanism?

Not really, the name is just assigned to the provides field, to do the checks, but the reverse is not possible, it is not possible to check for matches, because all internal dictionaries, and equality operations are based on the package name, as commented above, this would be a very big challenge.

[samuel-emrys]

Thanks for the detailed reply @memsharded, really appreciate you taking the time to engage on this.

• Why? Because when pkgb/1.0 and pkga/1.0 are built, they do not know that they depend on pkgc/1.1, but they build against pkgc/1.0. There is nothing in the profile, themselves, or their dependencies that tells them to build against pkg/1.1, so they resolve to pkgc/1.0. But app/1.0 needs binaries compiled against pkgc/1.1 otherwise it will have link errors, so app/1.0 fails with missing binaries for those dependencies.

Shouldn't this be resolved when the dependency graph is being constructed? My thought was that this is where the conan client provides the visibility over the entire chain, and tracks that there's an override from pkgc/1.0 to pkgc/1.1, so that the dependency tree when calculated becomes app/1.0->pkga/1.0->pkgb/1.0->pkgc/1.1, meaning that the build order calculation becomes pkgc/1.1, pkgb/1.0, pkga/1.0, app/1.0.

The way I would expect this to work is that overriding (in the safe context) would trigger a re-evaluation of the tree and resolve all overrides to a coordinate in the valid solution space. What I mean here, is that I would expect that you can only override to a version that lies within the intersection of all version ranges in the dependency graph (in the case of version). There's also an argument for an override in the unsafe context, in which you can override to a version outside of the intersection, with the caveat that here be dragons. Perhaps these two are being conflated in the language we're using here.

I would also expect that this is how lockfiles work - they are a specification of a coordinate in the solution space of combinations of versions/settings/options (I think this has been collapsed down to just versions for conan 2?) for each package and would generally always require tree computation. I think there's also value in being able to specify a coordinate outside of the valid solution space (without guarantees that it would lead to a meaningful tree) for users who have information that can't be captured by conan's dependency model.

The opinionated solution was previously the Conan one, that clearly deviated from the known good practices around versioning and versioning standards. At least for "maturity" level, the pre-releases are quite a standard approach across different technologies.

I agree with you that package maturity is perhaps better represented as a version (my opinion), however I disagree with your assessment that the previous defaults were more opinionated. Imposition of "best practices" is definitionally an opinion, because what "best practices" are is subjective and will change as standards and technologies evolve and if you provide only one way to achieve a thing, then you won't be robust to the changing landscape.

Whilst you or I may not consider it "best practice", It's a common practice to use different release channels to represent software maturity, especially where projects have not opted for semver. I'm a fan of semver - this isn't an attack, I'm just stating that this decision means that if they use conan, they would need to shoehorn semver principles into their package versioning to achieve a similar distinction between maturity levels because conan is coupling this to versioning scheme where it doesn't necessarily need to be.

I've seen your statements with regards to this refer to the removed ability to perform package promotion between channels (using conan copy). I think this is a complementary but orthogonal requirement and I think they can be detached. It can be simultaneously true that:

1. Promoting packages between release channels with conan is very difficult and therefore not supported because it's at odds with the guarantees you want to provide around package immutability
2. Packages can support release channels

The resolution here is that packages that use release channels have to be built multiple times - there's no native promotion process, and I think that's fine. It's a small inefficiency, but it's fine - that's the tradeoff with the package immutability guarantee. Especially because in an experimental release channel, you would expect temporal differences between the packages anyway (i.e., not every experimental release will make it to a stable release - this would only be the last package in a series).

Also, it is important to highlight that Conan 2.0 hasn't removed the overrides, they are still very possible and the test in #13745 proves it keep working. They are possible in recipes, it has been removed from CLI args, because that makes it yet another level of un-traceability, because those values are no longer capture anywhere. They do not exist in profiles, conf, recipes, lockfiles... they only exist in the caller (CI), which makes things more complicated to track, reproduce by developers, etc.

Appreciated - traceability is an issue. I would be extremely happy if overrides made it to lockfiles. As described above, if you think about a lockfile as the specification of a coordinate in the solution space of the dependency tree, this makes it the ideal place to perform a package override. I take your point that it isn't "that simple" if changing the dependency tree doesn't impact something like the package id to trigger a rebuild of the dependencies the depend on the overwritten one.

This is because we have saw many users during the last years struggling with using channel for this purpose, while a more standard versioning approach with pre-releases, version ranges, etc, seems to have better results in the majority of cases.

Yes, I can appreciate the difficulty that there's been in establishing workflows around this. It took me a while to work out how to approach it as well. In my mind it seems like there is a place for both though. To illustrate, I've had a chance to experiment with your suggestions and they've been really useful - so thanks for that. I think that they resolve most of my immediate use case, however there are a few things to highlight.

First is that when I'm tinkering with a recipe or a bit of code, it doesn't necessarily constitute a new release, but my experimentation. To address this in the past, I would normally export this as pkg/version@user/testing using the testing channel so that it doesn't conflict with:

• pkg/version@_/_ for CCI packages
• pkg/version@user/stable for stable packages
• pkg/version@user/experimental for development packages

Not conflicting here is important, because I want to be confident that pkg/version@user/{stable,experimental} refers to a build that the CI has performed. If I make modifications to these locally, it becomes very easy for me to lose track of what the actual release is and what my own changes are, which will lead to spurious and difficult to debug errors if I forget that I've made those changes and not cleared them from the cache. Now, mapping this over to your solution, we get:

1. pkg/2.0.0@user for stable packages
2. pkg/2.1.0-dev@user for development packages
3. pkg/2.1.0-test@user or pkg/2.0.0-test@user for test packages (testing could occur on any version, maybe to isolate a bug - not tied necessarily to the development process - this is an experimentation process)

I like the first two, but I don't like the third so much. This has the following implications:

• The version attribute can't be set. It needs to be set via set_version() instead, where the default version is read from a VERSION file or similar. This enables creation of the package with conan create . --user=user (uses the version in VERSION) or with conan create . --version=2.1.0-test --user=user, which creates pkg/2.1.0-test@user
• pkg/2.1.0-test@user takes precedence over both pkg/2.1.0-dev@user and pkg/2.0.0@user. This is good in the moment that I need it, but is something that I don't necessarily want automatically evaluated - I really only want to inject this on demand.
• pkg/2.1.0-dev@user takes precedence over pkg/2.0.0@user. This is the behaviour that I want, but I probably only want it for one or a small subset of the packages in the dependency tree. If the tree is comprised mostly of our internal packages, then I don't want the entire tree to evaluate to the -dev (or -test!) variant. I'd like this to be the exception, not the rule, and only really used when performing co-development.

To address this, we have core.version_ranges:resolve_prereleases. Unfortunately this has to be set in global.conf - it can't be set for an individual invocation of conan for quick switching, and from memory core confs can't be set for individual packages? I might be wrong on the second point there though.

Additionally, this only really works for in-source recipes. For out of source recipes, where a conandata.yml is present to define the versions, only the versions specified in conandata.yml can be used. This makes conan create hdf5/all --version=1.14.0-test not possible.

I could do something like conan create hdf5/all --version=1.14.0 --user=user to create hdf5/1.14.0@user, but I can't override the dependency tree to use this at the moment. This is useful where I want to experiment with a package on CCI, or of i want to fork the package to maintain it with features that for whatever reason can't be merged back into CCI. The ability to do this is actually important where I also want to use armadillo/11.4.3 for example, which has a dependency on hdf5/1.12.2, but I don't want to maintain a fork of armadillo/11.4.3 as well (as you've stated, I realise that I can embed this within the conanfile with requires("hdf5/1.12.2@user", override=True, force=True) - it would still be really nice to be able to quickly test this by overriding in a lockfile or similar method).

A completely different issue is the provides thing, when different packages (by name) can provide exactly the same interface and functionality. We have considered this problem a couple of times, but the "reference by name" is so core to the whole dependency resolution, that it would require a major complete re-write of most of the Conan core, not possible.

Fair enough. I see it as a generalization of the same concept, but that of course has no consideration for the implementation details, so if it's not workable it's not workable. Still, given that the requires("hdf5/1.12.2@user", override=True, force=True) syntax/feature is supported, I think that I would derive tremendous benefit from being able to achieve the same through a lock file or similar, rather than modifying the conanfile for the reasons above.

This has gotten quite lengthy and I'm not 100% sure I've been perfectly clear about the gaps that I still see. If you think there's value in breaking this out into discrete use cases, I'm happy to do so, just let me know.

[samuel-emrys]

I just thought that I'd also add, the practical barrier that I see at the moment is that given the following lockfile:

$ cat conan.lock
{
    "version": "0.5",
    "requires": [
        "sound32/1.0#83d4b7bf607b3b60a6546f8b58b5cdd7%1675278904.0791488",
        "matrix/1.0@user",
        "matrix/1.0#905c3f0babc520684c84127378fefdd0%1675278900.0103245"
    ],
    "build_requires": [],
    "python_requires": []
}

matrix/1.0@user is not interpreted as an identity for matrix/1.0#905c3f0babc520684c84127378fefdd0, so matrix/1.0 gets used instead of matrix/1.0@user.

Beyond that, I think that there are some useful helper functions that could be added the conan lock interface, but this would be the core functionality required to move this forward as I see it.

[memsharded]

Shouldn't this be resolved when the dependency graph is being constructed? My thought was that this is where the conan client provides the visibility over the entire chain, and tracks that there's an override from pkgc/1.0 to pkgc/1.1, so that the dependency tree when calculated becomes app/1.0->pkga/1.0->pkgb/1.0->pkgc/1.1, meaning that the build order calculation becomes pkgc/1.1, pkgb/1.0, pkga/1.0, app/1.0.

[samuel-emrys]

For tracking, relates to #9097