Update the lock file without upgrading dependencies

[cjolowicz]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Please provide a way to update the lock file without upgrading dependencies.

After adding a [tool.poetry.extras] section to pyproject.toml, Poetry displays the following warning, for example on install:

Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

That's fine, but if I run poetry update it upgrades my dependencies, which is not what I want at this time. If I run poetry lock instead, it still upgrades dependencies. Am I missing something?

Here are the relevant files and the commit:

• pyproject.toml
• poetry.lock
• commit

Sorry for not providing a smaller reproducible example, it's quite tricky to generate a poetry.lock file with outdated dependencies.

[finswimmer]

Hello @cjolowicz ,

have you tried running poetry install after poetry lock?

Which version of poetry do you use?

fin swimmer

[cjolowicz]

Hi @finswimmer, thanks for your reply!

I just reproduced this with Poetry 1.0.0b6. Originally I was on 1.0.0b4 when filing this issue.

To reproduce, clone my repository and check out b778b99. Invoke poetry lock, then git diff. The diff shows that Poetry added an [extras] section to the lock file (as it should have), but also upgraded several packages (pyparsing 2.4.4 -> 2.4.5, pytest 5.2.2 -> 5.3.0, pytest-mock 1.11.2 -> 1.12.1, xdoctest 0.10.2 -> 0.10.3).

I don't understand how running poetry install after poetry lock has any bearing on this. My issue is that there is no method to synchronize the lock file with pyproject.toml without also upgrading dependencies.

If I run poetry install after poetry lock, as you suggest, it installs any upgraded packages into the virtualenv. The modifications to the lock file (including dependency upgrades) remain, as would be expected.

[cjolowicz]

I just found a way to "trick" Poetry into doing this: Invoke poetry update foo where foo is some dependency that is already up-to-date. In my example repository, this would be poetry update click. This will add the [extras] section to the lock file and update the metadata content hash, without upgrading any dependencies.

Update:

A better workaround is provided below by @seansfkelley, see #1614 (comment). For some bikeshedding, I generally use this with insecure-package, a dependency unlikely to appear in any project.

poetry add insecure-package && poetry remove insecure-package

[finswimmer]

Sorry, it was my misunderstanding about what poetry lock is doing. It actually build the dependency tree based on the pyproject.toml from the scratch and doesn't take into account what's already in the lock file.

AFAIK at the moment there is no way to sync between pyproject.toml and poetry.lock. I thought there was already a discussion somewhere about this. But I couldn't find it.

Maybe two ways to go at the moment:

1. If you really need to stick at the currently used versions, write them as versions constrains in your pyproject.toml

or

2. Identify the newly added packages in the pyproject.toml, remove the corresponding lines and use poetry add to add them again.

[absassi]

The trick by @cjolowicz to run poetry update foo (which is a sign that code to recompute the lock file without updating packages is there, it's just that there is no command to run it) seems to work quite well.

However, on some cases, it loses markers and extras of dependencies that were not "updated". One option is to pass all those dependencies in the same update command, if possible (i.e. if they are already up-to-date). Otherwise, one has to be careful and discard these unexpected changes in poetry.lock.

[seansfkelley]

In #496 (comment) there is also outlined an alternate workaround: poetry add pathlib2 ; poetry remove pathlib2, where pathlib2 is any library you don't already depend on and that has no dependents... like pathlib2. (Strictly speaking, it can be any library with a dependency tree disjoint from your project, but why drag in more stuff than you need?)

I use this workaround all the time. Updating metadata in the lockfiles, resolving merge conflicts, etc. All of these operations get me into a state where I want something like poetry lock --preserve (which doesn't exist, but has a nice ring to it): re-locking, but keeping as many versions as possible where they currently are.

[PavlosMelissinos]

This functionality is much needed imo. A single package addition/update shouldn't trigger a dependency graph resolution for all the dependencies of the project.

I'd rather poetry lock/poetry update be more conservative by default. If we want to keep the current functionality as well, even though I consider auto-updates to be a bad practice, there could be an additional flag like --refresh to force irrelevant dependencies to update as well if there's a newer, compatible version.

[eblume]

I've been hitting this issue myself as we roll poetry out to a project with many developers in a relatively heterogeneous development environment. We are finding a lot of merge conflicts, particularly when dealing with dependency changes in a release/CI flow where merge conflicts can be a real frustration.

I've been trying to think through what a poetry lock --preserve might look like, as suggested by @seansfkelley. I wonder if we could have --preserve consider currently-installed versions of all targeted packages (even sub-dependencies) as implied version specifiers. In some cases, where the currently-installed packages may have been munged in some way (perhaps by a user manually running pip install while in the virtual environment), I suppose it could result in improper version locking or even a failure to lock when a "pure" (non---preserve) lock would have worked properly. I think it should be possible to detect this case, and message to the user in the failure message, something like:

Failed to write poetry.lock: The following packages are inconsistent with pyproject.toml:

• foo [ Specified as <TARGET>, currently installed: <VERSION> ]

You may wish to re-lock these packages without --preserve, or you might run poetry update <package> for each of these packages.

I may try to take a stab at a PR adding such a feature, but would love to hear more feedback on what --preserve might mean or if we want to to go with @PavlosMelissinos 's idea of changing the default behavior and adding a --refresh or something else.

[seansfkelley]

I'd rather poetry lock/poetry update be more conservative by default.

Fully in favor of this, however, the reason I suggested --preserve is that I figure this constitutes a breaking change and would therefore have to be delayed until 2.0.

By analogy, --remove-untracked was added to poetry install to make it more conservative, and then I filed #2370 to track making that conservatism default in 2.0.

I wonder if we could have --preserve consider currently-installed versions of all targeted packages (even sub-dependencies) as implied version specifiers.

I don't think this makes sense. The lockfile's job is to be the source of truth for what dependencies should be installed, if you noodled your virtualenv with pip install I think Poetry is well within its right to (and in fact, I claim, should) stomp on those changes and make your virtualenv match the lockfile.

This reminds me of a similar question though, which is how stringent should --preserve be? There are cases where you can't add/upgrade a package without an existing package also being updated. IIRC Yarn will make those updates, it just tries to do the minimal set of them. I think it would be impractical to interpret preservation as requiring no changes to other existing dependencies.

[PavlosMelissinos]

Fully in favor of this, however, the reason I suggested --preserve is that I figure this constitutes a breaking change and would therefore have to be delayed until 2.0.

By analogy, --remove-untracked was added to poetry install to make it more conservative, and then I filed #2370 to track making that conservatism default in 2.0.

Oh right, yeah that makes sense. Thanks for chiming in!

(On a side note, I don't suppose there are any critical situations where someone would prefer more changes to dependency versions, it's usually the other way around, so if it were up to me I might not consider it a breaking change and be less conservative than wait for v2.0. I'm not a huge fan of semver anyway, changing a number should not absolve developers of design sins nor allow backwards compatibility related misdeeds but I digress...)

[earshinov]

How to use it? poetry lock --no-update?

[abn]

@earshinov yes, but it is bot yet available in a released version.

[tony]

If you're on poetry 1.0 (tested with poetry 1.0.10)

You can also skip the poetry remove pkg1 pkg2 and just do poetry add pkgname, or poetry add 'pkgname@constraint' 'pkgname@constraint' for multiple packages at once, etc and it will update your packages in place. It also preserves comments and order in the pyproject.toml

Assume you already have a pyproject.toml. Skip poetry remove and just do:

Example:

poetry add \
   'asgiref==>=3.3.2,<4' 'django@~3.2.0[argon2]' \
   'django-nested-admin@~3.3.0' \
   'django-redis@~4.12.0' \
   'django-enumfields@2.0.*' 

Your poetry.lock file will be perfectly chiseled to just what you want.

Without this, the default behavior betrays the premise of lockfiles - keeping sub-dependencies steady unless you ask for it.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.