New 2.0 lockfiles

[memsharded]

Conan 2.0 will implement completely new lockfiles, with a new proposal of format, definition, behavior, usage and flow, based on the following principles:

• Lockfiles files will be json files containing only three lists of ordered references. One list for “host” requires, another list for “build” requires, another list for “python” requires.
• Lockfiles will no longer contain information about profiles, settings, options (as they did in 1.X)
• Lockfiles will not contain information about the dependency graph, only ordered lists (ordered by version and revision timestamp) of package references.
• The default level of locking will be locking down to the recipe reference, that is, including the version and the recipe revision (pkg/version@user/channel#recipe_revision), but not the package-id nor the package-revision. This is aligned with the previously accepted Tribe proposal of removing the package_revision_mode. Even if implementing lockfiles locking down the package revision will be possible, that will be considered the exception, and the main flows, documentation and behavior will be optimized for locking down to the recipe revision.
• Locking by default will be non-strict, that is, if some requires cannot find a matching locked version in the lockfile, it will be resolved normally. A --lockfile-strict mode will be implemented, but not the default, to enforce finding a match for declared requires in the lockfile or failing otherwise.
• A single lockfile file can lock multiple configurations, for different OS, compilers, build-types, etc., as long as they belong to the same graph. Lockfiles can be constructed for these multiple configurations incrementally, or they can be merged later. The concept of “lockfile bundles” will no longer be necessary
• Lockfiles will not be a version definition mechanism, they need to be a “realizable” snapshot of a dependency graph that should be able to be evaluated in the first place. However, they will allow their usage to define overrides or definitions of versions or recipe revisions that will be used if they fit in the valid definition of the original requires (that is, if the version fits in the version range of the recipe requires, or always for recipe revisions)
• Lockfiles will be allowed to evolve and adding new information to them easily, at conan install, create, graph, and export operations, specifying a --lockfile-out=newlockfile argument. That will allow evolving lockfiles when changes are done to the graph, while keeping control on those changes.

NOTES:

• The full rendered view of this proposal can be found in https://github.com/conan-io/tribe/blob/a851c4295da3ea1e7f4e7fa8a77f4aa039bc7c5b/design/034-new_lockfiles.md, it would be nicer to read than the raw markdown
• This proposal is large, and contains different aspects. It might be useful to have the discussions in review threads, rather than just responding. Try to use the Github review if possible
• Recall that released 2.0-alpha.6 implements a proof of concept of this proposal, so feedback from actual hands-on evaluation and experimentation would be very useful

---

• Upvote 👍 or downvote 👎 to show acceptance or not to the proposal (other reactions will be ignored)
  • Please, use 👀 to acknowledge you've read it, but it doesn't affect your workflow
• Comment and reviews to suggest changes to all (or part) of the proposal.

[sztomi]

This seems like a great proposal.

One list for “host” requires, another list for “build” requires, another list for “python” requires

Does this mean we can model and lock hidden requirements as "build" requires? (e.g. for an auxiliary binary package)

A single lockfile file can lock multiple configurations, for different OS, compilers, build-types, etc., as long as they belong to the same graph. Lockfiles can be constructed for these multiple configurations incrementally, or they can be merged later. The concept of “lockfile bundles” will no longer be necessary

Doesn't this contradict "Lockfiles will no longer contain information about profiles, settings, options"?

the main flows, documentation and behavior will be optimized for locking down to the recipe revision.

Perfect 👍

[a4z]

I am a little bit sad that you want to remove profile/option infos from the lockfiles

syncing profiles to developer machines can be also pain-point.
either you have redundant info, and add them into projects, or you use conan config install, what people forget to do or update ...

In fact, my additional wish would have been to even add the enabled remotes into lockfiles,
so they contain all infos, what recipe:revision used, where does a recipe:revision come from, and how was that created for a build (host /build profiles). (pin build/target/lockfile together, all info there)

with this proposal, it seems, my wish is gone for ever, and I wonder, if this will not have some negative impact to reproducibility or will require additional work on project site

[DoDoENT]

I don't quite understand this one. Isn't the whole purpose of lockfiles to be strict and prevent any changes such as overrides?

[memsharded]

Strict enforcement of the lock is most likely the single most reported issue in our support, and the origin of many frustrations. Recall that many operations imply in many occasions that it cannot be strict:

• Capturing a lockfile for every configuration after the first one, when the first lockfile from the first configuration is provided, to guarantee it is as consistent as possible between configurations. So if we have to capture a lockfile for N configurations, that means that N-1 conan lock create commands will have to add --lockfile-no-strict
• Every change that every developer does to a package, can potentially change the dependencies. Add a new one, update the version of a existing one, etc. This is a very common operation in the day-to-day. This also requires --lockfile-no-strict otherwise each developer change to requires will raise an error.
• Application of a lockfile created with conan lock add to lock only some of the dependencies or transitive dependencies will also require --lockfile-no-strict, always, by definition.
• The application of a partial lockfile, like the one used to build the changes done to a package in the middle of the graph, when we want to apply it to a downstream consumer application, will also require --lockfile-no-strict, always, by definition

One of the first implementation we had was the other way around, strict by default. When we realized that we were basically either saying strict=False in code (for some commands), or passing --lockfile-no-strict in the command line for the majority of command lines we had in our testing and examples (note that we have been working in more elaborte examples like conan-io/examples2#7, to further learn and evaluate this proposal), we decided to change the default.

Note that being non-strict doesn't mean that it will not lock things correctly. Whatever was locked will be correctly locked, unless the developer explicitly wanted to change dependencies outside of that lock. Lockfiles will prevent against any changes in the dependencies, if new versions or revisions are published. They never intended to be (and they are not in other language package managers) a mechanism to block developers to do modifications to source and recipes, and this was probably the root source of the above mentioned frustrations.

[ericriff]

My intuition tells me that lockfiles should be honored 100% of the times to ensure reproducible builds.

[mec-irobot]

If I understand correctly:

• If a package is referenced in the conanfile (or in some dependency in the graph) and something that matches that portion of the reference is found in the lockfile, the behavior is the same in strict and non-strict mode: the package used will match what is in the lockfile. In other words, for packages found in the lockfile, the lockfile is respected.
• If a package is referenced in the conanfile (or in some dependency in the graph) and something that matches that portion of the reference is NOT found in the lockfile, in strict mode the operation will fail, and in non-strict mode it will fall back to using normal conan package lookup mechanisms.

Is this a correct interpretation @memsharded ?

[memsharded]

Yes, perfect interpretation.

And this is the reason why at the moment we have proposed to be "non-strict" by default. If there is something in the conanfile that is not in the lockfile is because the users did a change to it, or is using just a partial lockfile or something like that, that is, because the user want that. Lockfiles are not a mechanism to prevent users doing changes to the graph, is a mechanism to avoid new published versions or revisions to affect users that didn't change their conanfiles, without them being able to control or block that.

[mec-irobot]

In light of that, would a conditional default for lockfile strictness make sense? For example:

• If --lockfile-out is on the command line, default to non-strict mode.
• If --lockfile is on the command line without --lockfile-out, default to strict mode.

Specifying --lockfile-out indicates that the user wants to generate a lockfile with new information, so non-struct makes sense. If no new lockfile is being generated, then strict may be a sane default as it is only consuming a lockfile and not writing a new one.

[DoDoENT]

So, basically, this is then the same as raw conanfile that does not use version ranges?

[memsharded]

No, it will at least also include the recipe_revision, which is very rarely included explicitly in conanfiles.
In this specific point, lockfiles don't deviate much of what they already do in 1.X. They are still "snapshots" of a real dependency graph, a specific instance in time of the recipes requires resolved to some specific values. But they cannot define a reality that cannot be produced by recipes in the first place.

[DoDoENT]

This itches me a bit. It means that the final resolved graph will be different than the one written in the conanfile.
If the installation fails here, it would immediately inform the developer that it has got a stale lockfile.

However, another option would be for this to automatically update the lockfile, so at least a diff would be seen during the committing to VCS.

It would be a really bad experience if you think that one state (i.e. the one from the lockfile) will be after issuing conan install ... and you end up with a totally different state, without knowing what happened (i.e. conanfile having precedence over lockfile).

[memsharded]

I think this has been elaborated in the above response: #34 (comment)

It is not possible to get to a different state using a lockfile, and the same starting point, lets say a given conanfile.txt. The different is if the conanfile.txt gets some changes that are no longer resolvable against the lockfile. As this happens to be an operation that happens very often in development, and users were frustrated about it raising an error, we made it non-strict as default. Of course if you want to get guarantees that this doesn't happen, that is the reason why --lockfile-strict is there, which you can always opt-in in CI to enforce things. But annoying users and blocking them to do local changes to their conanfiles was really a support burden.

[ericriff]

The non-strict behavior seems fine for development and testing, but if this is the default behavior I'm sure someone from my team will commit changes to the conanfile that "worked for me" but won't push and updated lockfile.

[DoDoENT]

OK, that makes sense. I understand now the reasoning why the non-strict mode was chosen as the default.

However, I think that in the above example with pkg/1.1 at least the lockfile needs to be updated automatically because without updating it we will end up with lockfile pointing to pkg/0.1, while the actual resolved package will be pkg/1.1. In this particular case, the conanfile and lockfile are incompatible (conanfile requires range [>=1.0 <2.0], and lockfile points to version 0.1) - this needs to be resolved somehow - either by warning the developer about the compatibility (maybe it does not need to be a hard error, like in strict mode) or by updating the lockfile so that the change would be visible in the lockfile diff when committing. It feels weird that conanfile takes precedence over locfile in cases like this.

I agree that CI could always use the strict mode, but it would be even better if the developer could be informed about the incompatibility between the changed conanfile and the existing lockfile as soon as possible.

[mec-irobot]

To ensure I understand correctly, if instead the conanfile was updated to:

 [requires]
 pkg/[>=0.0 <2.0]

Would your example command work and build with pkg/0.1 as described in the lockfile even if pkg/1.1 existed?

 conan install . --lockfile=conan.lock  # will work

[memsharded]

Yes, exactly. It will still lock to pkg/0.1.

[DoDoENT]

I argue that the strict mode should be the default and give non-strict option to developers. See my previous comment.

[mec-irobot]

Maybe it makes sense to allow the default to be set in conan.conf, so this can be propagated using existing using conan configuration synchronization mechanisms and have a sane default for different use cases?

For a group actively developing a large set of conan packages and using a complex CI mechanism with a lot of package changes being made by experts in Conan, non-strict is a great default.

On the other hand, if a group is primarily developing a large complex application that uses conan packages but most developers are not conan experts, strict is a better default.

[memsharded]

I don't think the criteria is the size of the application. Our rationale is more on the line of UX:

• By default don't annoy developers being strict, or forcing them to add --lockfile-no-strict, or to configure something different with a conf that will not match what is done in the servers (recall that conf can be synced and installed with conan config install, in theory to match what the CI is doing). Let them do the changes that they want. In any case they don't have publish permissions to the servers, only CI should be able to upload packages. They won't break anything for not being stricts.
• The CI is scripted. They know well which operations they want to be strict and what other can be not-strict, depending on their flow and process definition. As it seems the majority of lockfile operations would require non-strict for the most common flows (see above New 2.0 lockfiles #34 (comment)), we have favored the non-strict.

[DoDoENT]

It would be great if it were possible to do both that in a single conan invocation, at least with profiles.

Something like:

conan lock create . --lockfile-out=conan.lock --profile windows-profile --profile linux-profile --profile ios-profile --profile android-arm64-profile --profile emscripten --profile macos-profile ...

[memsharded]

This will be easily implementable with the custom commands and the public API at your will.

Note that your example, for a start is already having a different behavior, as multiple --profile arguments in CLI actually compound and collapse into a single one. So a new syntax would be necessary to define such different profiles. Multiply that by 2 (host and build profiles), and by 3 for (settings, options, conf, that are also part of the profiles), and we have made the conan lock create CLI arguments a nightmare, that would probably not satisfy many users.

But with the custom commands you can perfectly define what you want, what arguments you will be providing, the behavior that you want, and using the Conan Public API, you will be able to code it in relatively few lines (and distribute your custom commands with conan config install too)

[DoDoENT]

So, essentially, this looks like all different conan v1.X lockfiles are now merged together into a single giant lockfile?

[memsharded]

No, the Conan 1.X lockfiles were basically unmergeable, because they contained a graph, with numbered nodes for each package.

Conan 2.0 lockfiles are just lists of package references, merging them yes, is basically merging the lists and re-sorting them, in a largest lockfile.

[DoDoENT]

OK, I get it. This is why I stated above that it looks like a regular conanfile that does not use the version ranges. Of course, it also captures recipe revisions which we still don't use as we find it easier to reason about making a new minor/bugfix version when fixing the recipe - it's more transparent and removes the need for lockfiles which we also didn't need to use.

Yes, this does mean that we have packages like OpenCV/4.5.5.4@company/stable instead of having 4 revisions of OpenCV/4.5.5@company/stable, but it's more transparent and easier to reason about than having recipe revision hash in the conanfile or by using lockfiles.

Of course, since Conan v2.0 lockfiles are so much simpler, it may now have a sense for us to switch to using recipe revisions, but we'll see about that.

All in all, you did great work with the new lockfiles proposal!

[petermbauer]

Not sure i already understand all implications of this approach but here are some first questions:

1. Considering a quite simple usecase: Using the qt package from the conan-center while ensuring that the same RREV of the package and all its dependencies is used to avoid issues when new revisions with incompatible changes are published.
   This will be covered easily by the new lockfile approach but requires that the developers add --lockfile=conan.lock whenever running conan install/create/... and in other places like IDEs which is cumbersome and error prone.
   Would it make sense to have a default name (could also be configurable) for the lockfile so it is automatically used when next to the conanfile.py/txt?

2. A combination of the two examples for "Multi-configuration lockfiles" would look like this:

class Pkg(ConanFile):
    settings = "os"
    def requirements(self):
        if self.settings.os == "Windows":
            self.requires("dep/[>0.0]")
        else:
            self.requires("dep/[>0.1]")

Let's assume the first conan lock create under Windows will resolve to dep/0.1 but the next one under Linux resolves to dep/0.2 so both versions of dep end up in the same lockfile but the information which version has been used under which platform has been lost. Does this require separate lockfiles to be on the safe side?

[memsharded]

@sztomi

Does this mean we can model and lock hidden requirements as "build" requires? (e.g. for an auxiliary binary package)

Yes, if you recall one of the previous Tribe proposal, the dependency graph will always be complete, containing all requires, and tool_requires. It will be able to skip fetching the heavy binaries, but at least the recipes that model what dependencies are involved will always be fetched and be part of the graph. That means that lockfiles will always be locking the full set of dependencies, and transitive dependencies, including tool_requires

Doesn't this contradict "Lockfiles will no longer contain information about profiles, settings, options"?

No, the lockfile itself does not contain information about profiles, settings and options. But it can contain different packages for different configurations, like contain in the same lockfile windeponly/0.1 and linuxdeponly/0.1. I will add some clarification to the text with this.

[memsharded]

Hi @a4z

syncing profiles to developer machines can be also pain-point.
either you have redundant info, and add them into projects, or you use conan config install, what people forget to do or update ...

At least there is a well known, documented and supported way to sync profiles across developers and CI machines, which has proven to work reliable.
Syncing lockfiles is completely open, there is no way at all to distribute, share, use them, and that introduces a ton of variability and unknowns how to do it. The more lockfiles that need to by synced, the worst, and using 1 lockfile per configuration multiplies the problem by the N number of configurations to be supported.

In fact, my additional wish would have been to even add the enabled remotes into lockfiles,
so they contain all infos, what recipe:revision used, where does a recipe:revision come from, and how was that created for a build (host /build profiles). (pin build/target/lockfile together, all info there)

One of the previous Tribe proposals agreed on the package immutability, which means that the revisions guarantee that the origin of the package is irrelevant. As lockfiles lock revisions, they will be able to resolve the package succesfully and correctly between multiple remotes.

Furthermore, the fact that lockfiles doesn't lock the remotes is a desired feature, given that package promotion between remotes is a known good practice and is already being followed by users. You can get a lockfile against ConanCenter, then use it to install later and store a copy in your own private server, then disconnect from ConanCenter (something many users do), and the lockfile will still be working correctly.

[a4z]

There are real world scenarios where getting a package from a remote like CCI is totally unwanted, or even prohibited.
But that happens, I have seen it more than once. Packages on dev boxes come silently from the CCI , instead of an internal server, and troubles are here.

and I do not know what you mean that it's a problem to distribute a lockfile, the lockfile is saved in git alongside to a project, and can exactly restore todays, or yesterdays state. You do not want to have that all the sudden, yesterdays state is created with a library from a different remote.

The profile information in the lockfile was absolute handy to have, especially in scenarios where global profiles are used (to share profile state/info) . Or how else would you get the options set saved over time in git ?

How shall a lockfile allone restore/resolve the package successfully if it does only have the recipe revisions, but not the info about env and option from a profile.
Do I understand here something total wrong ?

[memsharded]

@petermbauer

Considering a quite simple usecase: Using the qt package from the conan-center while ensuring that the same RREV of the package and all its dependencies is used to avoid issues when new revisions with incompatible changes are published.
This will be covered easily by the new lockfile approach but requires that the developers add --lockfile=conan.lock whenever running conan install/create/... and in other places like IDEs which is cumbersome and error prone.
Would it make sense to have a default name (could also be configurable) for the lockfile so it is automatically used when next to the conanfile.py/txt?

We have thought about it, but we went finally with the "explicit is better than implicit". The thing is that having a "conan.lock" file there, that can easily be generated by some commands, and be automatically used, generates the opposite "I want to be able to install, but don't pick the default lockfile", so that would require a --lockfile-no-pick-default, which sounded like a poor CLI design and UX.

But the question is very valid, lets thing about this, see what other people think, I will also discuss with the team.

Let's assume the first conan lock create under Windows will resolve to dep/0.1 but the next one under Linux resolves to dep/0.2 so both versions of dep end up in the same lockfile but the information which version has been used under which platform has been lost. Does this require separate lockfiles to be on the safe side?

The idea is that the process of capturing a lockfile for different configurations result in a final valid lockfile for all of them, but not necessarily exactly the same along the process.
In your example, when you capture the lockfile for Windows, we assume that 0.2 has not been published yet, otherwise, it will resolve to it. So the lockfile contains [dep/0.1]. Now lets assume that 0.2 is published just before we create the lockfile for Linux, then we do the conan lock .. --lockfile=conan.lock --lockfile-out=conan.lock -s os=Linux, then the final lockfile will contain [dep/0.2, dep/0.1]. But that is perfectly fine, if you now apply the lockfile to installing, creating, the Windows configuration will resolve to dep/0.2 dependency, but it is a valid one for the range and it will be locked, and robust against further dep/0.X releases. But the fact that when you locked the first Windows configuration to dep/0.1 doesn't mean that you don't want to have the most homogeneous and updated dependency accross all configurations when you start applying the final conan.lock to the different Windows and Linux configurations. Both will resolve to dep/0.2, which makes more sense than the Windows one resolving to dep/0.1 and the Linux one resolving to dep/0.2, just because this one was published while you were computing the lockfile.

Does this make sense?

[ericriff]

I think leaving the package_id out of the lockfiles opens the doors for unwanted behaviors.
We use the lockfiles to achieve reproducible builds, particularly, to "ignore" new packages pushed to our repo unless we explicitly ask for them by updating our lockfiles.
Consider lib_a, which has a neon option set to False by default. If someone builds this package with neon=True and pushes it to our repo then our builds will start using a completely different package. Currently the lockfiles prevent this since the package_id is of each package is taken into account.

[memsharded]

Actually the current 1.X implementation doesn't substitute the package_id, it is also computed, and then validated.
There is a mode in the proof of concept that locks down to the package_revision, that includes the package_id. In the proof of concept it is only used to lock the package_revision, which is the real moving part, but I guess that validation of the package_id might be possible too, it seems adding "strict" check there is doable.

Consider lib_a, which has a neon option set to False by default. If someone builds this package with neon=True and pushes it to our repo then our builds will start using a completely different package. Currently the lockfiles prevent this since the package_id is of each package is taken into account.

In your case, someone creating a package with neon=True and pushing it to the repo, will not make it being used by consumers, unless consumers and profiles explicitly change to neon=True too. I think for the vast majority of cases this won't be an issue in practice. In the very extreme case, it would be possible to store a copy of the original profiles together with the lockfile, to be robust against possible future changes in the profiles.

[mec-irobot]

(I work with @ericriff FYI)

I think forcing the whole set of packages used to be locked to recipe_revision level addresses this.

We have had issues in the past where someone updates a package recipe (but not the version) and then builds that with a different option, which breaks building of older versions of our application. In our top-level application conanfile.py we specify all packages down to recipe_revision, but inter-package dependencies do not have a recipe_revision.

For example:

app/1.0 -> pkga/0.1@user/channel#RREV1 -> pkgb/0.1@user/channel. In this case app/1.0 specifies the full rrev of pkga, and pulls in pkgb implicitly with its default options (IE neon=False). Another app developer wants to use pkgb with neon enabled, but also fixes a minor issue in the recipe of pkgb when doing so. That developer also updates pkga to use pkgb with the neon=True option, and generates at least a new rrev if not also a new version (either way does not change this example). When we build our packages, we build the most recent version of all recipes and upload the binaries to our artifactory conan server.

We now have binary packages pkgb/0.1@user/channel#RREV1 with neon=False and pkgb/0.1@user/channel/RREV2 with neon=True.

The developer updates the conanfile for app/1.1 to build with pkga/0.1@user/channel#RREV2, and builds of that are happy. However, now builds of app/1.0 fail: when resolving pkgb it resolves to the most recent rrev of pkgb/0.1@user/channel#RREV2, but there is no binary of that package with neon=FALSE which is what pkga/0.1@user/channel/RREV1 requires.

We have many developers working on parallel on app from branches of various ages including release branches a few months ago, developer branches made a week or two ago, and bleeding-edge builds. Suddenly having a large portion of our developer community unable to build older code because we updated a second-order dependent conan package caused us a few big headaches, and turning on lockfiles for the app fixed that issue.

I think this current plan of simply locking all direct and indirect references down to recipe_revision is sufficient to address this issue for us, but I wanted to describe it in detail so we could all consider it.

[memsharded]

Yes, that is my understanding. Lockfiles capture the direct and transitive dependencies, so it doesn't matter if some other teams create a new recipe revision that only builds with a certain option, because that will not impact at all, that new revision will not be resolved by the first developer/team.

A different discussion of course is whether building only a subset of binaries, for example, only neon=True for RREV2, and not neon=False, knowing that it will disrupt developers that might upgrade to get the latest RREV2 recipe improvements. In principle, I would tend to favor the creating of binaries when source changes. If recipe gets a new RREV2, that should be built and green for the different configurations, or otherwise rejected. Because the thing is that it is not only that there is no pre-compiled binary for neon=False, but that change might completely break that option, and when downstream consumers don't find the binary and do --build=missing that build will fail without solution, that is problematic.

But I know it is a balance or trade-off between different things, so in any case, I think it will be good with the lockfiles always locking the full dependency graph down to the recipe revision.

[ericriff]

My intuition tells me that lockfiles should be honored 100% of the times to ensure reproducible builds.

[ericriff]

Can you be a bit more explicit on what a version definition mechanism means?

[memsharded]

Yes, that means, that you cannot have a conanfile.txt containing a [requires] dep/0.1, then capture a lockfile for it, then edit the lockfile to contain dep/0.2, apply it to conanfile.txt and expect that it will resolve to dep/0.2.

Lockfiles cannot be use to define a dependency, they can only be used to lock the moving or missing pieces of a requires to a fixed value. In most cases that means a lockfile can pin an exact version of a dependency in a given version range, and a lockfile can pin a recipe revision. But the definition of the dependency in conanfile requires should always be respected.

[ericriff]

I like this. Currently we delete and recreate the lockfiles each time we need to change 1 (or more dependencies). I'd like them to be able to change organically. poetry (python package tool) does a great job with this.
We maintain a huge, ever growing C++ project which uses conan only to consume its third party dependencies. We have lockfiles to ensure the builds are 100% reproducible, now in the past and in the future. What I mean is, we expect to be able to checkout a 6 months old commit, and as a consequence have a 6 months old lockfile and then be able to build that commit using the exact snapshop of dependencies we used at that point in time.
The problem we currently have is that when we're developing a new feature and add a new dependency or modify an existing one on our conanfile then we need to nuke our lockfiles and create new ones. As a consequence we update many packages from the lockfiles and not only the one we want (usually our conan remote moves faster than out codebase). I'd like to see a command that allow us to partially update a lockfile, like add a new dependency or change some version without affecting the rest (as much as possible).

[memsharded]

Yes, this is one of the biggest pain points at the moment in 1.X. And a lot of this proposal is intended to reduce or eliminate it.
The basic conan lock create command, can receive a --lockfile=xxxx argument, to use it as a starting point, then produce the output with --lockfile-out. That includes a lockfile from a previous point in time, from a subgraph, or from a different configuration. The resulting output lockfile will be the updated previous one, the things that can be locked from the previous one will be respected, the things that have changed or are new, will be added to the lockfile. To remove legacy, no longer used dependencies in the lockfile, the --clean argument can be provided too.

[ericriff]

The non-strict behavior seems fine for development and testing, but if this is the default behavior I'm sure someone from my team will commit changes to the conanfile that "worked for me" but won't push and updated lockfile.

[memsharded]

@a4z

There are real world scenarios where getting a package from a remote like CCI is totally unwanted, or even prohibited.
But that happens, I have seen it more than once. Packages on dev boxes come silently from the CCI , instead of an internal server, and troubles are here.

But this can be easily controllable with conan remote remove xxxx. Or by conan config install with a config that contains remotes.txt. Isn't the inverse even worse? If a lockfile is captured that points to ConanCenter, then the risk of fetching packages from ConanCenter because a lockfile says so, even that remote is not defined/enabled in the current client sounds very problematic.

and I do not know what you mean that it's a problem to distribute a lockfile, the lockfile is saved in git alongside to a project, and can exactly restore todays, or yesterdays state. You do not want to have that all the sudden, yesterdays state is created with a library from a different remote.

If you put the lockfiles inside a repo, and the conanfile.py for that repo is using for example scm, then every commit will generate a new recipe revision. Which means everytime you are adding a lockfile for a new configuration, it is inducing a new recipe revision for that package that will only contain 1 binary. But you will not be able by definition to have N binaries associated to the same recipe revision, resulting in a completely broken approach. The only possible way is to store the lockfile elsewhere, or at the very least be able to capture a single lockfile for all maintained configurations, before creating the different binaries, which is exactly what this proposal is describing.

The profile information in the lockfile was absolute handy to have, especially in scenarios where global profiles are used (to share profile state/info) . Or how else would you get the options set saved over time in git ?

The configuration of settings and options (and other stuff) will still come from profiles. The need to maintain, share and use profiles doesn't dissapear, because they are necessary in the first place. So instead of having multiple profiles named profile_win_vs15, profile_linux_gcc11, etc and lockfiles called lockfile_win_vs15, lockfile_linux_gcc11, the problem is now reduced to have the same profiles (you cannot just not define the profiles, they are necessary), and a single lockfile file. All you need is to do conan install --lockfile=conan.lock --profile=profile_win_vs15 instead of conan install --lockfile=lockfile_win_vs15, the problem complexity doesn't change, it is still the same, just reducing by N-1 lockfile files.

This proposal is about decoupling responsibilities:

• Configuration definition comes always from profiles
• Lockfiles exclusively implement the locking of versions (within version ranges) and revisions (within a version).

[a4z]

But this can be easily controllable with conan remote remove xxxx. Or by conan config install with a config that contains remotes.txt.

I mentioned, that does not scale and people manage to have wrong configs, in restricted environment, with a CCI as default, a real problem.

Isn't the inverse even worse? If a lockfile is captured that points to ConanCenter, then the risk of fetching packages from ConanCenter because a lockfile says so, even that remote is not defined/enabled in the current client sounds very problematic.

This is not the problem I have, nor I ever had. The only remote config should be the internal artifactory. urls within there may vary, but having this over time going with the code, e.g. switch from testing to stable packages, is wanted

If you put the lockfiles inside a repo, and the conanfile.py for that repo is using for example scm,

This theoretical scenario never effected my workflows, usually I know exactly what I want. No version ranges, no alternatives, always strict, in fact, I would love to see package revisions kept, but I understand that this is a problem.

The profile information in the lockfile was absolute handy to have, especially in scenarios where global profiles are used (to share profile state/info) . Or how else would you get the options set saved over time in git ?

The configuration of settings and options (and other stuff) will still come from profiles. The need to maintain, share and use profiles doesn't dissapear, because they are necessary in the first place. So instead of having multiple profiles named profile_win_vs15, profile_linux_gcc11, etc and lockfiles called lockfile_win_vs15, lockfile_linux_gcc11, the problem is now reduced to have the same profiles (you cannot just not define the profiles, they are necessary), and a single lockfile file. All you need is to do conan install --lockfile=conan.lock --profile=profile_win_vs15 instead of conan install --lockfile=lockfile_win_vs15, the problem complexity doesn't change, it is still the same, just reducing by N-1 lockfile files.

libcurl uses openssl on linux, and no openssl on Mac, how shall they have the same lockfile when they use different packages?
There are more examples like that.

• Configuration definition comes always from profiles
• Lockfiles exclusively implement the locking of versions (within version ranges) and revisions (within a version).

I agree that everything should come from profiles, but what is with command line arguments that add setting to conan (lock) create ?
They where added to the profile info in the lock file Now this info will be gone!
You can not restore/keep that info without throwing additional tooling onto the project and bake something.

And I usually need them in one place. That was awesome in the current lockfiles, except, only I missed the remote.
I am not sure you realize how much you destroy, I can not explain my workflows in detail, but I see that as a real problem.
Please keep the profile info in the lockfiles.

[memsharded]

@ericriff

The non-strict behavior seems fine for development and testing, but if this is the default behavior I'm sure someone from my team will commit changes to the conanfile that "worked for me" but won't push and updated lockfile.

I understand this is a concern, but please see comments in #34 (comment). If developers need to add --lockfile-no-strict for many of their commands, and in CI it is also a very common thing, then, devs will probably complain more and be more frustrated, while adding a --lockfile-strict in CI, in certain controlled flows, under your team defined practices, is very straightforward, and can provide you with the guarantees you want, without excessively annoying developers.

[vdsbenoit]

Great rework here, it sounds like a better approach in terms of developer experience 👏

With this new locking system, could conan somehow generate the lock file automatically upon conan install and conan create commands ? I mean, having the same behavior as npm i, which update package-lock.json at every run. Then, if a developer wants to reproduce the exact same build as the current commit, he/she can do it with a npm ci-equivalent command, which would read the lockfile.

With such a feature, the developer would not need to mind about lock files most of the time (i.e. handling conan create lock commands and --lock / --lockfile-out arguments).

Some people might not want to have this behavior by default. It could be behind an option in conan.conf

For those who are not familiar with npm: Difference between npm i and npm ci

[memsharded]

Hi @vdsbenoit

With this new locking system, could conan somehow generate the lock file automatically upon conan install and conan create commands ? I mean, having the same behavior as npm i, which update package-lock.json at every run. Then, if a developer wants to reproduce the exact same build as the current commit, he/she can do it with a npm ci-equivalent command, which would read the lockfile.

Yes, this might be possible, as those commands already have the --lockfile-out argument. At the moment they will raise, if you don't provide an input --lockfile first, but this relatively easy to enable. We were discussing this while implementing the proof of concept and we finally initially kept the initial conan lock create flow, because it is the most straightforward one when you have more than 1 profile/configuration to maintain. That is, we have considered the multiple configurations scenario the default one.

Under a multiple configuration scenario, it is better to first lock all possible configurations, as fast as possible, and ideally in just 1 machine, no need to distribute the command to different machines to capture this lockfile. This is possible with a conan lock create command as it doesn't install the binaries, so it should be relatively fast, and should be able to run repeateadly for different configurations in any machine.

But I agree that specially for the developer experience, it might make sense to enable the lockfile output directly from conan install command, one step less for developers. Lets see what other think, and I will bring it to the team too.

[petermbauer]

@petermbauer

Considering a quite simple usecase: Using the qt package from the conan-center ...

We have thought about it, but we went finally with the "explicit is better than implicit". The thing is that having a "conan.lock" file there, that can easily be generated by some commands, and be automatically used, generates the opposite "I want to be able to install, but don't pick the default lockfile", so that would require a --lockfile-no-pick-default, which sounded like a poor CLI design and UX.

I have poetry and pipenv in mind here as examples. The respective .lock file is used by default since somebody put it there on purpose and not using it or modifying it are more advanced tasks performed not that regularly.

But the question is very valid, lets thing about this, see what other people think, I will also discuss with the team.

thx!

Let's assume the first conan lock create under Windows will resolve to dep/0.1 but the next one under Linux resolves to dep/0.2 so both versions of dep end up in the same lockfile but the information which version has been used under which platform has been lost. Does this require separate lockfiles to be on the safe side?

The idea is that the process of capturing a lockfile for different configurations result in a final valid lockfile for all of them, but not necessarily exactly the same along the process. In your example, when you capture the lockfile for Windows, we assume that 0.2 has not been published yet, otherwise, it will resolve to it. So the lockfile contains [dep/0.1]. Now lets assume that 0.2 is published just before we create the lockfile for Linux, then we do the conan lock .. --lockfile=conan.lock --lockfile-out=conan.lock -s os=Linux, then the final lockfile will contain [dep/0.2, dep/0.1]. But that is perfectly fine, if you now apply the lockfile to installing, creating, the Windows configuration will resolve to dep/0.2 dependency, but it is a valid one for the range and it will be locked, and robust against further dep/0.X releases. But the fact that when you locked the first Windows configuration to dep/0.1 doesn't mean that you don't want to have the most homogeneous and updated dependency accross all configurations when you start applying the final conan.lock to the different Windows and Linux configurations. Both will resolve to dep/0.2, which makes more sense than the Windows one resolving to dep/0.1 and the Linux one resolving to dep/0.2, just because this one was published while you were computing the lockfile.

Does this make sense?

I understand but the lockfile then does not fully cover the "freeze the dependencies as they are" use-case. To be 100% sure one would need to maintain separate lockfiles for such an "exotic" dependency tree.

[DoDoENT]

I mentioned, that does not scale and people manage to have wrong configs, in restricted environment, with a CCI as default, a real problem.

I think that "restricted environment" and "CCI as default" shouldn't be together in the same sentence 😃 . As @memsharded already mentioned, in restricted company environments you should distribute your custom config to all your developers that has CCI removed. We do that and our developers cannot accidentally fetch any package from CCI. If they need a package from CCI, it first gets imported into our private Artifactory, possibly after recipe modification (this is necessary as most recipes in CCI are not able to build a mac/ios universal binary that we usually put in our packages).

This theoretical scenario never effected my workflows, usually I know exactly what I want. No version ranges, no alternatives, always strict, in fact, I would love to see package revisions kept, but I understand that this is a problem.

Well, this is also what we do - no version ranges, no package revisions, and, therefore, no need for lockfiles - in either old or new form. Why do you then feel you will get affected by this change? My company most surely won't be, but nevertheless, I still think this is a good change.

libcurl uses openssl on linux, and no openssl on Mac, how shall they have the same lockfile when they use different packages?
There are more examples like that.

The exact same example has been described in the proposal and what will happen.

I agree that everything should come from profiles, but what is with command line arguments that add setting to conan (lock) create ?
They where added to the profile info in the lock file Now this info will be gone!
You can not restore/keep that info without throwing additional tooling onto the project and bake something.

As far as I understand, those will now be encoded within the single lockfile. So instead of having separate lockfile for each profile, you will have "all those lockfiles" merged in a single lockfile. Also, please see this conversation.

[a4z]

DoDoENT That you are in a now perfect environment is nice for you.
If you bring something new to a place, and the wrong person burns the fingers, the new might not arrive and the state you are talking about will never be reached. and this is my problem with the default setup, convenient for a desktop user, but for company you need to start with rolling out configs. I see that as a problem, since I have been there more than once.-

The difference about

conan install . --lockfile=conan.lock -s os=Windows -s:b os=Windows
conan install .  --lockfile=conan.lock -s os=Linux -s:b os=Linux

and

conan install . --lockfile=conan_windows.lock 
conan install .  --lockfile=conan_linux.lock 

is of course ...
and a more cumbersome way to create lockfiles since it requires more stages, basically updating existing profiles ..

Maybe when I see that in action, my mind will adopt and change, that is usually what happens.
And I am not against that in general, even if I have a problem to map that new behaviour in my mind to existing workflows.

I usually have huge trust in the conan team that they do the right change.
And if there is some easy alternative to preserve the profile information that was used to create the pipleine..
This is information that was available so far, tied to a lockfile, I got it for free, I rely on it, and that will go away, and that is a super braking change for me.

[DoDoENT]

@a4z , I see what you mean and I understand.

However, when comparing

conan install . --profile windows

and

conan install . --lockfile=conan.lock --profile windows

It's much more consistent than using conan install . --lockfile=conan-windows.lock.

Yes, it's a super breaking for cases like yours. But it will make general experience more consistent. And Conan v2.0 is the breaking release anyway (new cache format, new toolchains and generators, mandatory lower-case package names (which I still don't like 😛 ), ...) so IMO it's better to make that change now than be sorry later.

[memsharded]

@petermbauer

I understand but the lockfile then does not fully cover the "freeze the dependencies as they are" use-case. To be 100% sure one would need to maintain separate lockfiles for such an "exotic" dependency tree.

Yes, of course, if for some reason you really want to have the Windows build use dep/0.1 and the Linux build to use dep/0.2, even if the later is perfectly valid in the range for the Windows one, then you need to maintain separate lockfiles.
Something similar happens with different applications/products, developed at different points in time. Lets say that you have app1/1.0 and app2/1.0 and they both depend on dep/[>0.0 <1.0].
Both scenarios are valid, depending on the business needs and development flow:

• If you want to have dependencies for both apps as consistent as possible, you can try to feed the lockfile of one of them into the other and capture a new lockfile, or even try to maintain in in a single lockfile. This means that when something changes while processing one of the applications, like one of the upgrading to a new version of dep (maybe because in some part of the dependency graph they require strictly dep/0.9), then, that would induce such a change in the other app.
• If you prefer to keep both apps dependencies isolated, and locked, you can and should maintain completely different lockfiles for each one of them. In that way, one app can be locking dep/0.1, and the other can be locking dep/0.9, and it won't be an issue.

Sometimes, when the apps are relatively coupled, like they are part of some larger systems, the first behavior might be preferred, while if the apps are completely unrelated products, the second is the way to go.

In general, one of the things that I like most about the new lockfiles is their conceptual simplicity: "an ordered list of package references", that will be used while resolving a graph. There is nothing intrinsic to them that force you to use them for just one configuration, one product, one version... You can perfectly feed one lockfile from one product/app into a different product/app, or you can feed it from one version to the next version, or from one configuration to a different configuration. Or you can keep them separated if that is what you want for your use case.

Note, however, that for the case described above and your ""freeze the dependencies as they are", I still think that having a single lockfile resolving to dep/0.2 is the correct solution. The "dependencies as they are" is not completely defined if we need to capture N different configurations, that can take us, lets say, 1 minute, and in that minute the dependencies can actually concurrently change. I find more correct to see the "dependencies as they are" when the N configurations are fully captured, that is, when the operation finalizes, not when the operation starts. The freeze is only complete when all configurations have been frozen, not when the freezing operation begins.

[mec-irobot]

Is this a new syntax to specifying recipe revisions? Can you describe how this was changed or provide a pointer to that document?

[memsharded]

The new syntax will probably be only used and visible in lockfiles, where something like pkg/version#recipe_revision%timestamp will be there, to allow the ordering of revisions.

Internally we had to change all the model of references in order to propagate together the timestamp with the revision, just for the lockfiles, but for normal operation of Conan nothing changes. The pkg/version#recipe_revision is still a full, deterministic coordinate of that recipe, it can be used as an input to commands, etc.

This is a great observation, because actually there is a case, the one that does conan lock add specifying a reference including the recipe revision, when other references for exactly the same version with their recipe revision already exist. In that case, it might be necessary to specify the timestamp if we want to define the correct ordering. I am creating a follow up issue for this, because this needs to add some testing, and there could be some UX to polish around this use case.

Update: new issue conan-io/conan#11129

[mrjoel]

In general the proposal looks good, better alignment and interplay between recipe and lockfile in particular.

I'll point out that many of our teams have a relatively small set of devs who are responsible for and familiar with the conan config and updates. For other devs we want to provide as few and concise of commands as possible to facilitate updating. They don't need awareness of conan nuances, especially having to remember to add extra flags to executions which are usually the default desired behavior. We and they just want the easiest path to obtain a known and deterministic state. It's not simply a discussion of boolean distinction between "local dev" vs "CI execution".

With that context in mind, I'll emphasize that it's important to consider the conan-consuming developers' UX experience at least as much as the conan-changing developers', in particular that the frequency of execution of commands to update recipe and lock files may be much less frequently used than commands to install or updated based on an existing lockfile.

Further I'll echo and agree with concerns raised and discussed elsewhere, that I'd like to see.

• Use lockfile by default if one is present under a default filename (conan.lock?)

  I have to imagine that the cases where a lockfile is present but not desired for use are far outnumbered by the cases of "use the lockfile because someone went to the trouble to put it there". This would leave --lockfile=foo as only needed when specifying an alternate lockfile name. For what it's worth, from my perspective the needed --no-lockfile option doesn't seem at all cumbersome or UX misaligned.

• Cleaner CLI with default lockfile

  I'll additionally note that using a default lockfile when present would also allow notable CLI cleanup. In particular the proposal provided example could be reduced as illustrated below, making it significantly less cumbersome. The --lockfile option would only be needed when not using the default, and --lockfile-out would be even less frequently used since it could default to the input lockfile, either the default or as specified with --lockfile. In other words, if not specifying a different output lockfile then assume in-place modification.

  conan lock create . --lockfile-out=conan.lock -s os=Windows -s:b os=Windows
  conan lock create . --lockfile=conan.lock --lockfile-out=conan.lock -s os=Linux -s:b os=Linux
  

  becomes

  conan lock create . -s os=Windows -s:b os=Windows
  conan lock create . -s os=Linux -s:b os=Linux
  

• non-strict behavior

  As others have expressed, I also have reservations and concerns about unilaterally specifying non-strict behavior. I think it may be reasonable as a default, but only as long as there is an option to specify strict mode as part of a configuration mechanism storable and shareable in VCS alongside the lockfile itself (i.e., don't have to remember the add the special option each time a command is executed). Would it be reasonable to allow specifying within a lockfile itself whether it should be handled in strict or non-strict mode, with the default empty meaning non-strict?

  My understanding is that in non-strict mode the lockfile will be updated if there is a mismatch, while in strict mode nothing will be changed and an error will be emitted? If so then this parallels discussions and approaches between the npm and yarn mechanisms, where yarn allows specifying --frozen-lockfile within a .yarnrc parallel to the package info files (or immutable in berry). I prefer that mechanism over needing to remember to use npm ci as a separate command from npm install.

  For many shops and projects like mine, that would allow us to specify strict mode, and only the smaller set of conan savvy devs need to remember to add a theoretical --update-lockfile option (commandline should override config, unlike yarn) and only then when running commands intentionally to bump dependencies.

[memsharded]

@petermbauer

Assuming that the most common usage pattern is mkdir build && cd build && conan install .. i suggest to take the conan.lock from the recipe dir instead of the cwd.

I already tried when doing PR conan-io/conan#11135, and I found these cases:

• There are cases where there is no conanfile at all, like conan install --requires=zlib/1.2.11. It would necessary to have a different logic: in this cases, the lockfiles are to be found in cwd, but in these other cases the lockfiles will be in the conanfile folder. Seems more consistent to be able to use cwd always.
• Consider the flow:

  mkdir build && cd build
  conan install .. --lockfile-out=conan.lock  # could be ``conan lock create ..``
  conan install .. 
  conan build ..

  In this case, the conan install .. --lockfile-out=conan.lock, the conan.lock file is generated inside the build folder. I think many users would be annoyed if conan decided to put the lockfile in the parent directory, polluting their project, so I think that conan.lock inside build folder is the expected behavior. Would you expect subsequent installs with conan install .. and other commands to use that conan.lock lockfile automatically or not? I think so, you just created it, it is in your current folder... if not you will

So it seems that cwd is a better default.

[mrjoel]

Assuming that the most common usage pattern is mkdir build && cd build && conan install .. i suggest to take the conan.lock from the recipe dir instead of the cwd.

I already tried when doing PR conan-io/conan#11135, and I found these cases:

...

So it seems that cwd is a better default.

I'll also offer my request and two cents towards considering having the lockfile default be alongside the recipe instead of cwd. By far the most common use case we encounter is recipe and lockfile tracked in source control, and an out of source build tree, with steps similar to the following. Some of these steps are often wrapped in a helper script run by devs instead of running the conan command directly.

git clone <repo> /some/path/to/src
mkdir /other/prefix/for/build
cd /other/prefix/for/build
... # non-conan build area configuration
conan install /some/path/to/src/{path_or_reference1,path_or_reference2,path_or_reference3}
...
cmake ...

As for the examples you offer, both seem more like corner cases where using --lockfile=specificfile.lock would be a reasonably expected pattern. In particular creating a lockfile in a build directory strikes me as a bit odd since if won't typically be shared with others from that location, but would still be possible by explicit specification.

[memsharded]

Assuming that the most common usage pattern is mkdir build && cd build && conan install .. i suggest to take the conan.lock from the recipe dir instead of the cwd.

@petermbauer @mrjoel

Fair enough, I think there are pros and cons, but your proposal also makes sense. So unless someone else thinks contrary, I am proposing these changes:

• The default conan.lock location will be besides the conanfile. That means, for consuming commands like conan install <path> the lockfile will default to in --lockfile=<path>/conan.lock (given it exists, it won't fail if it doesn't exist). For conan lock create <path> it will also create <path>/conan.lock unless --lockfile-out explicitly defines other location
• For commands that don't use a conanfile, like conan install --requires=pkg/version, it will use --lockfile=<cwd>/conan.lock as default (given it exists)

I will also try them first in code in the PR, then when that PR is merged, I will come back to change this proposal.

[ytimenkov]

So unless someone else thinks contrary, I am proposing these changes: The default conan.lock location will be besides the conanfile

That makes more sense to me when lockfiles don't contain profile information and can work with multiple configurations, in contrast to v1 lockfiles (when it's per-configuration then it's better to keep it in build folder).

Also, I got impression that it's safe to update conan.lock, i.e. use same file in both --lockfile and --lockfile-out, is it correct? Then it's one more argument to keep it next to the recipe, because one can potentially quickly and easily create lockfile for multiple configurations.

[memsharded]

Also, I got impression that it's safe to update conan.lock, i.e. use same file in both --lockfile and --lockfile-out, is it correct? Then it's one more argument to keep it next to the recipe, because one can potentially quickly and easily create lockfile for multiple configurations.

yes, it is safe to update conan.lock, so indeed a conan lock create -pr=profile1 && conan lock create -pr=profile2 will be enough for creating a lockfile for both profiles, no need extra arguments.

[KerstinKeller]

Some of the previous comments seem to evolve around the following questions:

What are lockfiles used for?

We use lockfiles to make dependency resolution deterministic e.g. reproducible.
This usecase seems to be preserved with the current proposal, especially with strict resolution being the default.
In this way, I think the proposal is great!

What it sometimes used for?

Some of the comments suggest (and actually this is also how I inteded to use them) is to preserve state.
E.g. to capture information about the Conan call itself (e.g. which profile, which options, ...) as some kind of Cache.
When you think CMake and it's invokation, you do have a cache.
E.g. you can do

mkdir _build && cd _build
cmake .. -G"My generator" -DSOME_OPTION=ON -DANOTHER_OPTION=OFF
cmake .. -DANOTHER_OPTION_I_FORGOT=ON
cmake ..

Where on the second / third invokation all variables from the first invokation are still in place.
For Conan, it's nothing the like. Calling conan subsequently with different arguements will consider only the arguments given.

I think that Conan would also benefit from having a cache, to make subsequent invokations easier.
However, lockfiles are not the proper way to store that cache, maybe this issue needs to be addressed separately (as @lasote already mentioned - though I am not sure if capturing all info given into a new profile will cover all usecases)

[memsharded]

Hi Tribe!

The PR conan-io/conan#11135 with the necessary changes from your feedback to the proof of concept has been merged (it will be part of alpha.7), and this proposal has already been updated to incorporate this feedback too, check the latest commit: 2b9b968.

As the capturing of combined profiles is mostly a UX improvement/requirement, but doesn't imply any functional logic to how the lockfiles work in 2.0, it doesn't need to get a solution right now in this proposal, and a new issue in conan-io/conan#11155 has been created for it, we will be working there to provide the best possible UX when the first hands-on feedback starts coming.

[rconde01]

Just trying to wrap my head around this...in the middle of trying to make things work with 1.X lockfiles...so I have some context. It sounds a bit like Conan 2.0 lock files will have some resemblance to Conan 1.0 lockfiles with the --base option? In fact I was going down the route of only trying to use the --base option, but then I couldn't specify the profile when installing (although seems like it could theoretically be possible?). It's not specified above, but with the new lock files it's valid to specify a profile?

[memsharded]

Just trying to wrap my head around this...in the middle of trying to make things work with 1.X lockfiles...so I have some context. It sounds a bit like Conan 2.0 lock files will have some resemblance to Conan 1.0 lockfiles with the --base option? In fact I was going down the route of only trying to use the --base option, but then I couldn't specify the profile when installing (although seems like it could theoretically be possible?). It's not specified above, but with the new lock files it's valid to specify a profile?

Yes, exactly. This is what we learned from Conan 1.X, we had to implement the --base approach because of high demand, and we learned that users were trying to use them widely to simplify things.

But you are right, they don't allow being used directly, they were designed as a base for the full lockfiles, but not for direct usage. It would be possible to implement, but it was not easy.

So yes, this proposal for 2.0 not only allows to specify profile, the new lockfiles need a profile to be specified, because they will only capture version and revision information, but not configuration. They idea is that commands always look the same, using or not using profiles:

conan create . -pr:h=linux -pr:b=windows
conan create . -pr:h=linux -pr:b=windows --lockfile=mylockfile
conan create . -pr:h=rpi -pr:b=windows
conan create . -pr:h=rpi -pr:b=windows --lockfile=mylockfile
conan install. -pr:h=linux -pr:b=windows
conan install. -pr:h=linux -pr:b=windows --lockfile=mylockfile
....

[rconde01]

It would be good to see more practical examples...some of it seems a bit abstract.

1. I set up my project with a conanfile.py
2. I want to lock the dependencies...so i create a lock file (and add it to git?)
3. In my CI I conan install using that lockfile?
4. Then later I want to update a single library...let's say from v1 to v2...what is my process? Or maybe i just want the latest recipe for the same version...what is my process?

[memsharded]

It would be good to see more practical examples...some of it seems a bit abstract.

Yes, the idea of these proposals are that they are mostly conceptual, architectural and strategic, deprecations, evolutions...
The implementation and the docs for 2.0 and all the beta process will be focused on polishing all these things from practical examples.

What you are asking is basically:

# add a "conan.lock" lockfile to the current project with conanfile.py or conanfile.txt
conan lock create .  # can specify here configuration, can repeat for multiple configs
# the lockfile can be added to git
git add . && git commit ...
# "conan.lock" is used by default
conan install .  # will use conan.lock in the repo to lock dependencies

This will work well for consuming. For creating packages it should be noted that commiting new files to the repo can change the recipe revision (when using scm coordinates, for example)

Evolving the lockfiles is possible, for example:

conan lock add --requires=mydep/1.1

Assuming the lockfile had previously mydep/1.0, and that requires in the graph accept that in their ranges (recall the proposal details about lockfiles not overriding), this will allow to evolve the lockfile to upgrade new dependencies.
Removing in the lockfile the mydep/1.0 and specifying --lockfile-partial will allow to re-evaluate the lockfile and resolve it again, to update (we haven't planned a conan lock remove argument, but is doable).

[rconde01]

Yes, the idea of these proposals are that they are mostly conceptual, architectural and strategic, deprecations, evolutions...
The implementation and the docs for 2.0 and all the beta process will be focused on polishing all these things from practical examples.

I understand...but practical examples also make for clearer discussion of the issues and potential problems/gaps.

Evolving the lockfiles is possible, for example:

Well perhaps conan lock update mydep/1.0 or conan lock upgrade mydep/1.0 mydep/1.1 would be nice.

[memsharded]

Well perhaps conan lock update mydep/1.0 or conan lock upgrade mydep/1.0 mydep/1.1 would be nice.

Yes, as I was saying, a conan lock remove might be possible too. conan lock add was mostly added as a proof of concept that this "manual" modification of lockfiles is doable. Such conan lock update you are commenting, the great thing is that it is basically a sed command over the lockfile, probably not a critical functionality to use the lockfiles, so this will come with real hands-on usage.

Because it is possible that for many cases, it is not even necessary. For example, when processing the changes that actually bump mydep/1.0 => mydep/1.1, it is possible, maybe even desirable, to process that bump new conan create to build mydep/1.1 with the lockfile, so you are building it with fixed dependencies. That same conan create can update the lockfile and add there mydep/1.1 directly, so it becomes automatically available for the downstream consumers that would be applied this lockfile.

[KerstinKeller]

One completely different though which has just popped up (which I didn't explicitly see before), is how this proposal relates to version controlling lockfiles.

E.g, at the moment we do have the following workflow:

• Our consumer projects work with lockfiles, so we don't accidentially introduce side effects when maintaining our recipes (revisions enabled) and break builds
• Lockfiles are versioned alongside conanfile.py for all consumer profiles that are relevant for the given project
• When introducing new dependencies, we create a branch, update the conanfile, potentially build new packages (if they don't yet exist in that combination), update the lockfiles and then create a PR.

With 1.x lockfiles can become quite hard to compare.
Merging lockfiles based on git stategies is impossible, and requires us to rebase branches in case that the conanfile.py and lockfiles have changed in between.

Are lockfiles meant to be version controlled? Are they meant to mergeable?

[memsharded]

@KerstinKeller

yes. This is how a lockfile looks like:

{
    "version": "0.5",
    "requires": [
        "pkgc/0.1#4d2acb63aa360295c54a1811799a5e87%1652111238.2041075",
        "pkgb/0.2#476b31358d78b3f04c68c4770bd6a79c%1652111239.1558757",
        "pkgb/0.1#43511a4669edc78292516e6cbf00c0b2%1652111238.0599482",
        "pkgawin/0.1#2f297d19d9ee4827caf97071de449a54%1652111237.7507892",
        "pkganix/0.1#2f297d19d9ee4827caf97071de449a54%1652111237.9206934",
        "app1/0.1#34d26743473acd7f8d8026502501c170%1652111238.346824"
    ],
    "build_requires": [],
    "python_requires": []
}

As they are ordered lists, one element per line, and always sorted, it is expected that they would be relatively easy to merge and version control.

Are they meant to mergeable?

Yes, because merging ordered lists is something that is possible. We implemented a convenience conan lock merge in our proof of concept, as it was convenient for testing the CI flows:

(conan2_36) λ conan lock merge -h
usage: conan lock merge [-h] [--lockfile LOCKFILE] [--lockfile-out LOCKFILE_OUT]

optional arguments:
  -h, --help            show this help message and exit
  --lockfile LOCKFILE   Path to lockfile to be merged
  --lockfile-out LOCKFILE_OUT
                        Filename of the created lockfile

The result of the merge doesn't guarantee a fully "clean" lockfile (it can contain not used dependencies versions/revisions) because it doesn't evaluate the dependency graph, but the merged lockfile can be perfectly used to resolve and lock dependencies, and can be --lockfile-clean at any time to trim obsolete dependencies.

[ytimenkov]

As they are ordered lists, one element per line, and always sorted, it is expected that they would be relatively easy to merge

Are they ordered or are they sorted? Sorted lists are indeed easy to merge, but for ordered it is impossible (at least in general case): For example, if 2 packages are in reverse order (e.g. one lockfile "downgraded")? This somewhat reminds about conflict resolution in SCM (as source code is an ordered list of lines :D ) which quite often is wrong.

[lasote]

@ytimenkov I didn't know there is a difference between sorted and ordered.

[ytimenkov]

Yeah... Got me 😕

What I apparently was thinking about is that "sorted" is according to internal properties, like lexicographical name or semver version. While "ordered" is according to external order, like "in order of insertion".

Dictionary says "sorted" is also grouped or "in certain order". While "ordered" means (to me) that you can't reshuffle things, more general.🤷‍♂️

[sztomi]

We implemented a convenience conan lock merge in our proof of concept

Small remark regarding that, it would be nice if this made it to the final version and it could take an arbitrary number of lockfiles to merge into one (i.e. nargs="*" for the input lockfiles).

[memsharded]

What I apparently was thinking about is that "sorted" is according to internal properties, like lexicographical name or semver version. While "ordered" is according to external order, like "in order of insertion".

The lists are sorted by version and revision timestamp, to account for the logical resolution within ranges, and revisions. Should be good for source control.

Small remark regarding that, it would be nice if this made it to the final version and it could take an arbitrary number of lockfiles to merge into one (i.e. nargs="*" for the input lockfiles).

Yes, it is multiple input :)

def lock_merge(conan_api, parser, subparser, *args):
     subparser.add_argument('--lockfile', action="append"