Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent Output When Locking PyYAML #6513

Closed
2 of 3 tasks
Kurt-von-Laven opened this issue Sep 14, 2022 · 8 comments
Closed
2 of 3 tasks

Inconsistent Output When Locking PyYAML #6513

Kurt-von-Laven opened this issue Sep 14, 2022 · 8 comments
Labels
kind/question User questions (candidates for conversion to discussion)

Comments

@Kurt-von-Laven
Copy link

  • I am on the latest Poetry version.

  • I have searched the issues of this repo and believe that this is not a duplicate.

  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

  • OS version and name: Ubuntu 22.04 LTS

  • Poetry version: 1.2.0

  • Link to the contents of your pyproject.toml file: pyproject.toml

Issue

Some time between yesterday (September 13th, 2022) at 13:39 UTC and today (September 14th, 2022) at 08:28 UTC, poetry lock --no-update began producing inconsistent output on the same pyproject.toml, breaking all of our CI pipelines. When run locally on Ubuntu 22.04 LTS, poetry lock --no-update continues to consistently produce this poetry.lock. This is consistent with past runs on Ubuntu 22.04 LTS in CI and hence what I perceive to be the correct output. Here is a CI workflow showing the diff (reproduced below) between the old (correct) and new (incorrect) output. The new output appears consistent across CI runs. Note also that two spaces were added before metadata.python-versions, which is the focus of #6201, not this issue.

diff --git a/poetry.lock b/poetry.lock
index f6fa86c..e255791 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -284,7 +284,7 @@ python-versions = "*"
 
 [metadata]
 lock-version = "1.1"
-python-versions = "^3.10.6"
+  python-versions = "^3.10.6"
 content-hash = "67141421e7355f09ced17905557c8a8fdf47fc58eaa23f7de2658e6797666990"
 
 [metadata.files]
@@ -402,6 +402,13 @@ PyYAML = [
     {file = "PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:f84fbc98b019fef2ee9a1cb3ce93e3187a6df0b2538a651bfb890254ba9f90b5"},
     {file = "PyYAML-6.0-cp310-cp310-win32.whl", hash = "sha256:2cd5df3de48857ed0544b34e2d40e9fac445930039f3cfe4bcc592a1f836d513"},
     {file = "PyYAML-6.0-cp310-cp310-win_amd64.whl", hash = "sha256:daf496c58a8c52083df09b80c860005194014c3698698d1a57cbcfa182142a3a"},
+    {file = "PyYAML-6.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:d4b0ba9512519522b118090257be113b9468d804b19d63c71dbcf4a48fa32358"},
+    {file = "PyYAML-6.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:81957921f441d50af23654aa6c5e5eaf9b06aba7f0a19c18a538dc7ef291c5a1"},
+    {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:afa17f5bc4d1b10afd4466fd3a44dc0e245382deca5b3c353d8b757f9e3ecb8d"},
+    {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:dbad0e9d368bb989f4515da330b88a057617d16b6a8245084f1b05400f24609f"},
+    {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:432557aa2c09802be39460360ddffd48156e30721f5e8d917f01d31694216782"},
+    {file = "PyYAML-6.0-cp311-cp311-win32.whl", hash = "sha256:bfaef573a63ba8923503d2[753](https://github.com/ScribeMD/docker-cache/actions/runs/3051597658/jobs/4920017184#step:7:770)0362590ff4f576c626d86a9fed95822a8255fd7"},
+    {file = "PyYAML-6.0-cp311-cp311-win_amd64.whl", hash = "sha256:01b45c0191e6d66c470b6cf1b9531a771a83c1c4208272ead47a3ae4f2f603bf"},
     {file = "PyYAML-6.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:897b80890765f037df3403d22bab41627ca8811ae55e9a722fd0392850ec4d86"},
     {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:50602afada6d6cbfad699b0c7bb50d5ccffa7e46a3d738092afddc1f9[758](https://github.com/ScribeMD/docker-cache/actions/runs/3051597658/jobs/4920017184#step:7:775)427f"},
     {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:48c346915c114f5fdb3ead70312bd042a953a8ce5c7106d5bfb1a5254e47da92"},
Error: Process completed with exit code 1.
@Kurt-von-Laven Kurt-von-Laven added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Sep 14, 2022
@dimbleby
Copy link
Contributor

dimbleby commented Sep 14, 2022

the new wheels were uploaded to pypi in that time period, as you can verify at https://pypi.org/project/PyYAML/#files.

I expect the difference in behaviour is cache related, presumably you locally have a cache that does not know about the recently uploaded files.

I'm not seeing a bug here.

@Kurt-von-Laven
Copy link
Author

Kurt-von-Laven commented Sep 14, 2022

We pin the exact version of PyYAML, and we are running poetry lock --no-update in CI. Are you saying PyYAML overwrote a release? We cache the virtualenv in CI as well and got a cache hit, so I don't understand what would be cached locally that wouldn't be cached in CI.

@mkniewallner
Copy link
Member

mkniewallner commented Sep 14, 2022

We pin the exact version of PyYAML, and we are running poetry lock --no-update in CI. Are you saying PyYAML overwrote a release? We cache the virtualenv in CI as well and got a cache hit, so I don't understand what would be cached locally that wouldn't be cached in CI.

New artefacts for an existing release can be added after a version is uploaded in PyPI. This is exactly what happened here, and this can be seen here, where I guess 3.11 support has been added, and wheels got uploaded for an existing release:
Screenshot 2022-09-14 at 14 48 11

--no-update from poetry lock means "don't update versions that don't need to be updated", but it will still add new artefacts if new ones are uploaded afterwards.

@Kurt-von-Laven
Copy link
Author

That all makes sense and is very helpful; thank you. If I understand correctly, I should run poetry cache clear pypi --all locally, and commit the proposed changes to poetry.lock? It seems like it should be possible for an attacker to exploit this behavior of poetry lock --no-update by adding malicious artifacts to an existing release without creating a checksum mismatch?

@mkniewallner
Copy link
Member

That all makes sense and is very helpful; thank you. If I understand correctly, I should run poetry cache clear pypi --all locally, and commit the proposed changes to poetry.lock?

If the newly added wheels look good to you, yes, clearing the cache locally should make sure they are discovered.

It seems like it should be possible for an attacker to exploit this behavior of poetry lock --no-update by adding malicious artifacts to an existing release without creating a checksum mismatch?

The hashes are calculated from the files that are uploaded, so depending on the algorithm, a new artefact with the same name would have a different hash (assuming that there is no possible collision for the hash function that is used).

@Kurt-von-Laven
Copy link
Author

Excellent; thank you.

I meant more that the content-hash and all other existing hashes still match, but I suppose anyone who wants to be using Poetry securely should be hashing the entire lock file themselves regardless.

@neersighted neersighted added kind/question User questions (candidates for conversion to discussion) and removed kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Sep 14, 2022
jayaddison added a commit to openculinary/crawler that referenced this issue Sep 24, 2022
See python-poetry/poetry#6513 for an explanation of the partial-update (additions) to pyyaml-related content hashes
@antdking
Copy link

antdking commented Nov 7, 2022

Given stale cache causes inconsistent locking, isn't this a bug?

We've just been chasing why our lockfile flipflops between 2 machines, and this seems to be it.

Workaround we're using for now is to run with --no-cache, though this results in over 5m runtime in our environment.

antdking pushed a commit to datapane/datapane that referenced this issue Nov 7, 2022
- disable poetry cache while performing locks, as cache invalidation doesn't seem to work still.

ref: python-poetry/poetry#5282
ref: python-poetry/poetry#6513
antdking pushed a commit to datapane/datapane that referenced this issue Nov 21, 2022
- disable poetry cache while performing locks, as cache invalidation doesn't seem to work still.

ref: python-poetry/poetry#5282
ref: python-poetry/poetry#6513
antdking pushed a commit to datapane/datapane that referenced this issue Nov 21, 2022
- disable poetry cache while performing locks, as cache invalidation doesn't seem to work still.

ref: python-poetry/poetry#5282
ref: python-poetry/poetry#6513
antdking pushed a commit to datapane/datapane that referenced this issue Nov 21, 2022
- disable poetry cache while performing locks, as cache invalidation doesn't seem to work still.

ref: python-poetry/poetry#5282
ref: python-poetry/poetry#6513
Copy link

github-actions bot commented Mar 1, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/question User questions (candidates for conversion to discussion)
Projects
None yet
Development

No branches or pull requests

5 participants