[question] How to use conan center with a specific version of conan client ?

[johan-boule]

We are facing some complications using the conan center repository for our CI builds.
For CI stability, we want to decide when we upgrade the conan client, so we usually don't have the latest version.
In order to use packages compatible with our version of conan, we enabled revision support and create lock files to depend on specific revisions of the packages, and also create a separate CONAN_USER_HOME for each build (i.e. for each branch).

While this is working, I must say this is very painful to maintain.

Indeed, whenever I want to add or upgrade a dependency, this is what I'm doing (perhaps by lack of better knowledge):

• Go to conan.io and look for the package I'm interested in
• For each revision, starting with the most recent (in the drop-down list order):
  • Look at the recipe to see the minimum required conan version
  • Double check it's indeed what it claims by looking at the imported python modules (often recipes claim a min required conan version, but in fact need a more up-to-date conan)
  • Put my browser in development mode to be able to get the revision hashsum as text and copy it in my clipboard (you can't select the text of the drop-down list)
  • Try to download/build a binary package with a random configuration (e.g. shared release)
  • Transitive dependencies will usually fail because they require a higher version of conan, so, for each of them:
    • Recursively redo all the steps described above!
• Once everything installs, generate a "base" lockfile to record the recipe revisions of all our direct and transitive dependencies
• Download/build binary packages for all the configs we need
• Upload all of them to our in-house repository partly because we don't want to take the risk of revisions disappearing from conan-center (and also because of security reason, and lack of proxied-repo with v2/revision support, we can't use conan center in our CI)

Is there some way of handling this which doesn't involve all those manual actions ?
Or else, does Conan 2 have plans to address this ?

I think we would be fine if the conan center repository could be addressed with some global revision number that would only make visible the recipes and binary packages that where available at that point in time, even if that prevent seeing new libraries.
Basically, we are fine with having the notion of a globally versioned repository, like classic linux distros do, which also has the big advantage of solving version conflicts.

Also, as a side question, since conan center stores binary packages, why doesn't it store source packages ? By source, I mean the whole source code of the libraries. We would be much more confident if our ability to rebuild everything from scratch didn't depend on "random" external websites and only on our in-house repo or conan center itself. Just like linux distros have their own copy of all the source code of all packages. An other reason is our enterprise network is configured in a quite complicated way which involves proxies and overridden certificates to allow access to each website. I often see conan trying to download from e.g. sourceforge or whatever and fail because it doesn't have the right enterprise certificate for that particular website (we have to synchronise its cacert.pem with the system certs). It would be much more simpler if we only had to setup the certificate/proxy for conan center only.
As a workaround, is there a way I can force conan to generate a true self-contained source package so that I can upload it to our in-house repo ?

I was pondering whether the question about source packages should be asked as a separate topic, but I think source and revisions are intertwined, or actually, should not be. My idealised repository structure would be something like this:

libfoo/1.0/upstream-source.tgz
libfoo/1.0/conan-1.50/recipe-revision-2.tgz # fixes some packaging bug in the recipe
libfoo/1.0/conan-1.50/recipe-revision-1.tgz # breaking change / modernisation of the recipe
libfoo/1.0/conan-1.45/recipe-revision-2.tgz # back port of packaging fix above
libfoo/1.0/conan-1.45/recipe-revision-1.tgz # first version of the recipe

Imagine I have conan 1.48 (so, neither 1.45 nor 1.50), unless I specify some particular recipe revision, I would expect it to download conan-1.45/recipe-revision-2.tgz (because presumably conan 1.48 can use recipes made for 1.45), and upstream-source.tgz (which is independent of conan's version or recipe)

In this pseudo view, I'm using revisions as incremented numbers, but conan uses some hash of course. How conan knows which revision is the latest is a mystery to me though.

[memsharded]

Hi @johan-boule

There are a few different questions here, lets try to address them:

• Regarding the ConanCenter evolution, there is this recent post: https://blog.conan.io/2022/09/20/consuming-recipes-during-migration-conancenter.html. I think you might be missing the most recommended, simple and robust way to achieve stability, is storing a copy of your used packages in your own repo. This is a very simple procedure, just install + upload. When you want to try to upgrade, you can install from conanCenter again, put those in another temporary server repo, and test things against it, until everything works, then copy that temporary server repo to the main one. Running your own server repo with a copy of the things you need in production is a best practice, not only for Conan but for all package managers

Regarding the lockfiles, it is true that 1.X lockfiles are complicated and challenging to master. Conan 2.0 is a huge improvement over that, it will allow 1 single conan.lock file for all configurations, that can be even committed to the repo and that is automatically used at conan install. Please try the 2.0-beta.3 release lockfiles and give feedback.

Also, as a side question, since conan center stores binary packages, why doesn't it store source packages ? By source, I mean the whole source code of the libraries

This is true that it would be convenient. We did some proof of concept in the past, and this is scheduled to be resumed after 2.0 (what we called the 2.X target): #8211

[johan-boule]

Thank you for the reply.

My post was a bit too long, and my main point perhaps got lost.

Thank you very much for smoothing the transition towards Conan 2. While we are still using Conan 1.x, we are in fact already using the forthcoming features of 2.x that have been backported to 1.x.

We are already using "RREV" lock files committed into our git repo (we didn't need to commit PREV locks so far). And we are already using our own conan repo to upload the recipes and binary packages we need. In fact, our CI doesn't have internet access and exclusively uses our in-house conan repo. We are also already using separate conan home dirs for each build/branch as is recommended, with a shared download-cache to avoid redownloading the same packages between builds of different branches when they are the same.
So, with this setup, our CI is stable and it's fine.

My issue is that, when I want to add or upgrade a dependency, on a dev box, I temporarily enable the conan center repo, and then everything falls apart because currently the conan client doesn't seem to have a mechanism to avoid trying to use recipes which are for higher versions of conan. It wouldn't make sense to upgrade the conan client on my dev machine, because I need to stick to the same version as our CI.

So I am hoping that Conan 2.x includes a plan to address this specific point.

As a workaround, I'm pondering whether we should make an internal mirror of Conan's Git repo so that our CI can pip install from it any version from a tag, and put it into a "venv" isolated inside the build directory. In other words, have a different conan version for each branch our CI is building. With that, I might "afford" upgrading conan just to add or upgrade a dependency and enjoy browsing conan-center without trouble :)

Alternatively, we will produce a new container image with an upgraded conan version every time we want to touch a dependency. Every build/branch will use a different version of the image. This might be the easiest route.

[memsharded]

My issue is that, when I want to add or upgrade a dependency, on a dev box, I temporarily enable the conan center repo, and then everything falls apart because currently the conan client doesn't seem to have a mechanism to avoid trying to use recipes which are for higher versions of conan. It wouldn't make sense to upgrade the conan client on my dev machine, because I need to stick to the same version as our CI.

So I am hoping that Conan 2.x includes a plan to address this specific point.

I am afraid this is not possible. Even if recipes have a conan_minimum_required, having to search the space for recipes that satisfy these constraints is a known NP-hard problem, that requires backtracking in the dependency graph when a node doesn't satisfy constraints. In practice, as for every hypothesis it is necessary to hit servers, download files, unzip them, load the python files, and evaluate a few methods, that makes the problem intractable.

Alternatively, we will produce a new container image with an upgraded conan version every time we want to touch a dependency. Every build/branch will use a different version of the image. This might be the easiest route.

Yes, in practice the problem is approached in the other way: first you upgrade your Conan version in CI and dev machines. We do a ton of effort, due to our stability commitment not to break things, so it should be safe in most cases to update it. Having a CI process to do this, and control which Conan version is used in prod is recommended. Then, once you have an updated Conan version, you can try to update to the latest ConanCenter packages.

[johan-boule]

Maybe I'm not entirely thinking this through, but I think the constraint on the conan version shouldn't lead to the same NP-hard problem as constraints between libraries: the version of conan is not an adjustable variable, it's a given constant.
I think the constraint can be quickly evaluated independently for each package as long as the network protocol makes it possible to retrieve the list of recipes and for each one a metadata describing the conan version constraint. Alternatively, we could imagine the conan client sending its version to the server, and the server doing the constraint filtering by itself.
If several revisions of a recipe fulfill the constraint, I don't know what metadata can be used to determine which recipe is the most up-to-date (as you can't tell what hashsum is "greater"), but conan.io seems to be able already to list the revisions from most to least recent.

[johan-boule]

Now that Conan 2 is out, is there any chance that you implement a server-side filter so that only the recipes made for a Conan version less than or equal to the client's version are returned ?

[memsharded]

I am afraid this is still not possible. This kind of server side functionality is very challenging to implement and maintain, it requires a full new version of the protocol, give time to the many vendors that implement the protocol to adopt it, etc.

Not to say that it would still be an NP problem, yes. When you get to a specific recipe that declares compatibility with a given Conan version, that specific recipe can contain requires() with version ranges and normal requires() that would resolve by default to the latest revision. Following the desired behavior, it would be necessary to explore the version-range and the revision full space, looking for a combination that satisfy the current conan version. When a violation of the support for the current version, further hypothesis of the dependencies will be explored. When the full space of combinations of the dependency graph (the sub-graph) is exhausted without a solution, it would be necessary to take the current package hypothesis, go back 1 revision (or 1 version, or both), and start again the full evaluation of the full space of the sub-graph of dependencies. That is NP, requiring backtracking down the dependency graph everytime a hypothesis violates support for the current client version.

So this is not realistically possible.
I know that the 1.X->2.0 transition requires some effort, but such a big major won't happen again. Otherwise, the recommended approach is continuously update the Conan client version to the needs of the dependencies. And if staying in a fixed Conan version is a strict requirement, then the approach would be to not allow the dependencies to use more modern client features. For users with these kind of requirement the recommended approach is to work with a fork of conan-center-index repo, not with the ConanCenter server directly. There are many users following this approach with great success, and very happy about the control, flexibility and robustness they achieve with it.

Thanks very much for your feedback, I am closing this ticket as not planned.