Poetry 1.2 lockfiles can't always be installed by Poetry 1.1

[tomwatson1024]

For version details etc, see repro below.

Issue

Poetry 1.2 can create lockfiles that aren't installable by Poetry 1.1 due to different handling of . characters in dependency names.

If the configured repository uses the "simple" repository API, Poetry 1.2 replaces . characters with - characters in the name field of dependencies in the lockfile. Poetry 1.1 fails to install from this lockfile with a SolverProblemError.

I'm not sure what guarantees, if any, Poetry makes about lockfile compatibility between Poetry versions. This isn't a "breaking" change, as such, but makes interoperability painful in cases where the Poetry version can't be upgraded "atomically". It might be that this just isn't supported, in which case a "fix" might be a quick note in the changelog or something.

I've got a Docker-based repro using ruamel.yaml:

# syntax=docker/dockerfile:1.3-labs
FROM python:3.8 as poetry1.2
RUN pip install poetry==1.2.1
COPY <<"EOF" /src/pyproject.toml
[tool.poetry]
name = "dotdash"
version = "0.1.0"
description = ""
authors = [""]

[tool.poetry.dependencies]
python = "^3.8"
"ruamel.yaml" = "*"

[[tool.poetry.source]]
name = 'simple'
url = 'https://pypi.org/simple'
default = true

EOF
RUN cd /src && poetry lock

FROM python:3.8 as poetry1.1
RUN pip install poetry==1.1.15
COPY --from=poetry1.2 /src/ /src/
RUN cd /src && poetry install

which produces:

 => ERROR [poetry1.1 4/4] RUN cd /src && poetry install                                                         1.9s
------
 > [poetry1.1 4/4] RUN cd /src && poetry install:
#15 0.995 Creating virtualenv dotdash-VsnhxLU2-py3.8 in /root/.cache/pypoetry/virtualenvs
#15 1.569 Installing dependencies from lock file
#15 1.706
#15 1.706   SolverProblemError
#15 1.706
#15 1.706   Because dotdash depends on ruamel.yaml (*) which doesn't match any versions, version solving failed.
#15 1.706
#15 1.707   at /usr/local/lib/python3.8/site-packages/poetry/puzzle/solver.py:241 in _solve
#15 1.732       237│             packages = result.packages
#15 1.732       238│         except OverrideNeeded as e:
#15 1.732       239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
#15 1.733       240│         except SolveFailure as e:
#15 1.733     → 241│             raise SolverProblemError(e)
#15 1.733       242│
#15 1.733       243│         results = dict(
#15 1.733       244│             depth_first_search(
#15 1.734       245│                 PackageNode(self._package, packages), aggregate_package_nodes

Ignore the non-standard install method for Poetry - I've also tested with the recommended method, I'm just being lazy in the Dockerfile.

[tomwatson1024]

See also #4242.

edit: As another friction point when interoperating Poetry 1.1 + 1.2.

[neersighted]

#4242 is unrelated. The names in the lock file should be interoperable as we specifically reverted to 1.1-compatible behavior after a beta which canonicalized names in the lock file. We should be writing the pretty name now, which should be compatible.

PTAL @radoering / @dimbleby when you have a minute (and if you could gist the lockfile @tomwatson1024, it would make things a little easier).

[tomwatson1024]

Can do - https://gist.github.com/tomwatson1024/90fbda6ddbffc501e65bf774e2af8d9e

[dimbleby]

lockfiles produced for packages from legacy repositories have normalized names - because the name here is normalized.

(The above configuration forces pypi to be treated as a legacy repository)

On master similar is also true for pypi proper, the name taken from the API here turns out to be the normalized name too.

I'm not sure this is worth worrying about:

• the cat is out of the bag at this point
• re non-legacy: does it matter if poetry 1.1 doesn't like poetry 1.3 lockfiles?

It would be possible for poetry to parse a non-normalized package name out of the wheel name at the same time as it parses the version. In principle there's no guarantee that this will give the answer that this issue hopes for, in practice it probably would.

[neersighted]

Ah, good eyes -- I was surprised by this as I remember we specifically opted out, but I missed how PyPI had been restated as a legacy source. I don't think we care at all about 1.1 and 1.3 lock files, so I think we can safely close this as a non-issue -- PyPI and 'legacy'/PEP 508 sources are treated differently for both technical and compatibility reasons, and if you force Poetry to violate that invariant things get a little messy.

[tomwatson1024]

PyPI and 'legacy'/PEP 508 sources are treated differently for both technical and compatibility reasons, and if you force Poetry to violate that invariant things get a little messy

Though the example above forces PyPI to be treated as legacy, I actually hit this issue with a private source that uses the legacy API. The unusual configuration for PyPI above is just to provide a share-able example!

[neersighted]

@dimbleby Any thoughts on if we can revert the 1.2 branch to 1.1-compatible names for legacy repositories? I believe it was you who last refactored them.

[dimbleby]

It would be possible for poetry to parse a non-normalized package name out of the wheel name at the same time as it parses the version. In principle there's no guarantee that this will give the answer that this issue hopes for, in practice it probably would.

Not seeing better ideas right now (but wouldn't be very keen on that one!)

It would maybe be easier to backport to 1.1 a fix to canonicalise names properly when reading from the lockfile.

I know that tomwatson1024 was having trouble with poetry 1.2.0 because of #6453, but 1.2.1 solves that. To my mind that models a better approach: understand what's stopping people from leaving 1.1 behind altogether, and sort out those problems.

[neersighted]

Hmm, I think that is fair. And I very much agree that the better path is to make sure everyone can migrate to 1.2, but I also think pragmatism is useful in stuff like this.

I'm split on the idea of backporting a change to 1.1 -- overall we might just want to document and wontfix this as it's generally easy to avoid (if you're using sources you are likely sophisticated enough to make sure everyone is on the same minor).

[vito-c]

I was unable to get my 1.2.x lock file to run on our build server that uses 1.1.x Is there any documented workaround? I tried reverting my poetry version to 1.1.x, deleting/resetting my lock file but now everything hangs at Resolving Dependencies...

[neersighted]

@vito-c That sounds like a different issue entirely unrelated to the one here. You likely have a damaged cache from interrupting Poetry -- try blowing away $(poetry config cache-dir)/artifacts and running poetry cache clear --all ..
In most cases, installing a lock file generated by Poetry 1.2 with Poetry 1.1 should work just fine; if you're running in to problems I would suggest joining Discord or starting a discussion so that others can help you determine what the root cause might be.

[DoodlesEpic]

Just wanted to add feedback here, this is still a problem. Consider the following use-case:

• I have Poetry 1.3.x installed in my local dev machine
• I have a CI server with Poetry 1.1.x

Right now I have the problem that I simply cannot use Poetry for the CI build and instead have to rely on exporting a requirements.txt and installing through pip. I believe that this kind of change in behavior should be reserved for major versions (i.e. Poetry 2.x) since it's indeed a breaking change.

I believe the simplest way to fix this for now is to release updates for 1.2.x and 1.3.x that revert this change in behavior until Poetry 2.x. Otherwise add a flag or configuration to one of the newer versions that allows generating the older lock-file.

[dimbleby]

I think that's unlikely to happen - the ship has sailed - and this issue probably ought to be closed to reflect that.

You should either not use poetry 1.1 in CI, or not use poetry 1.3 locally. (That's the same position that you would have been in if the poetry version had gone to 2.0; except perhaps you'd have felt less annoyed about it)

[DoodlesEpic]

You should either not use poetry 1.1 in CI

If I could I would. The specific example I gave is real, I was attempting to deploy a previously working application on https://render.com/ (a PaaS similar to Heroku). I do not control the Poetry version in the CI server and it just so happens that they are still on 1.1.

or not use poetry 1.3 locally

Admittedly this would indeed work, but then it also requires that anyone else that ever decides to install those dependencies also use the outdated Poetry version so they don't break the lock-file. As you said, if it was a major version this wouldn't be a problem at all, just having it packed in a minor update is in my opinion a bug and not a desired change.

I agree that there has been a long time already and the change is somewhat solidified, that's why I suggest at least setting up a flag for this to make sure any environments with similar limitations don't have to go through the frustration of constantly exporting requirements.txt files and installing through PIP.

Anyway, just my thoughts, if no one plans on working on this I guess the issue should indeed be closed as it's pretty easy to workaround.

[neersighted]

Version numbers are merely marketing here -- if we had bumped the major, like @dimbleby said, this situation would be identical. You would have to use an old version of Poetry that is not receiving fixes to maintain compatibility with an older version of Poetry.

Also, this issue was tracking differences in behavior around the complex interaction between the lock file and the solver across versions; your concerns center solely around the changing of the lock file version, where no backwards compatibility is promised.

In any case, I think that this entire issue is stale due to time. We should probably formally document the versioning we are using (read: Python-ish with incremental change, unlikely to bump major unless the CLI UX experiences a breaking change), but that would be another issue. If you want to open it, I encourage you to :)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.