Use === to install lockfile versions #3250
Comments
|
Hey @ryanhiebert, We did a quick check and this issue looks very darn similar to
This could be a coincidence, but if any of these issues solves your problem then I did a good job 😄 If not, the maintainers will get to this issue shortly. Cheers, |
|
@ryanhiebert Please specify the source in the Pipfile: django-cleanup = {version = "*", index="aspiredu"}and relock |
|
#3248 is about it writing the incorrect index to the wrong, and it does work to manually specify the index in the This issue is about the fact that uploading a new and local version to a custom index that's already in the Pipfile will cause previously generated lockfiles to make builds that are broken. This happens because even though the lockfile specifies an exact version on PyPI, and includes checksums from the PyPI index, the version with the local part ( If it is the position of the maintainers of Pipenv that the solution is to specify the indexes for the package in the I find this solution unsatisfying, because by needing to explicitly indicate which index the package should come from in the I hope that clarifies the purpose of this issue. |
|
@ryanhiebert I setup a private pypi as you described and do some testing as follows:
So the only failed case is 1, 2, 3. The key point here is PyPI recognizes ConclusionI understand your requirement here and I recommend two workarounds:
Thanks. |
|
Thank you for the workaround, it will likely be a way to avoid this issue until a better solution is found, though it is not an ideal solution in the end, because as far as I can tell, both by reading the text of the PEP and the regular expression, it doesn't appear that those names are valid version identifiers from PEP 440. Re-locking or updating before install is not reasonable. I'm using Pipenv in a CI environment, and I've got to be able to rebuild the dependencies from the lockfile, or else I simply can't trust Pipenv in my builds. It seems to me that perhaps using arbitrary equality matching would be most appropriate for lockfiles. It completely avoids this issue, and the special case of local version identifiers is explicitly mentioned in the PEP. Unfortunately, it appears that the authors of the PEP may not have been considering lockfile use-cases when they were writing it, because they have some very strong recommendations to prefer not using arbitrary equality. https://www.python.org/dev/peps/pep-0440/#arbitrary-equality @dstufft or @ncoghlan : As the authors of that PEP, do you think that Pipenv would be better served by using arbitrary equality when installing package versions from the lockfile, to avoid issues arising from local version identifiers being compatible with identifiers with the same public version? |
|
Yes, using PEP 440 specifically calls that out as a possible use case:
The admonition in the paragraph immediately after that one could be softened to also allow for the hash-checked lockfile with pre-normalised version identifiers use case. |
|
Thank you, @ncoghlan , that's what I suspected that paragraph was intending, but the wording of the admonition to avoid using prompted me to want to check on that. Thank you for confirming that. |
ryanhiebert
changed the title
|
Yeah this is spot on actually, but it does make me wonder whether we should then avoid normalizing names we write to the lockfile? For example, if we asked pypi directly for this kind of version number, would we get a response? I’m pretty sure we would. Not 100%, but it’s just a matter of checking. I feel like we should avoid normalizing this, so the question is just, where is the normalization occurring? I’m also pretty sure |
|
I just spent 20 minutes trying to find where the version is written out from the lockfile, and I just got lost, I wasn't able to figure out how to make it write the version verbatim using |
|
@techalchemy From the PyPI side, the server doesn't know why you're asking for a particular version string, so it always performs the normalisation when looking for a project version: https://github.com/pypa/warehouse/blob/1fbb4ac752e68b5840b9e09b68e44a165569bfa6/warehouse/packaging/models.py#L135 This means that while the text on this front is currently ambiguous in PEP 440 itself, I think it would make sense to propose a clarification that says that for PEP 440 compliant strings, normalisation is still taken into account, and it's only non-compliant strings that are handled as being completely opaque. The only parts that won't happen are the parts which are part of the definition of the |
|
So to summarize the basic proposal is to, for example, switch to using |
|
@techalchemy That sounds like a pretty safe option to me, but there's one corner case you're going to need to consider closely: the zero-padding case, where you can specify a version like I think it should be fine, since generating the lock file necessarily gets the version to pin from the found artifact (not from the original request in |
This is the kind of distinction that makes me really happy to have access to you, there is a 99% chance I would gloss over it without it specifically being pointed out. I suspect I'll at least ignore it for now, but at least I'll do it consciously :) Thanks! |
|
This has been solved with my work on restricted package indexes last year, so I don't think we need to support a |
Issue description
We have a private package registry that we use to ship temporary versions of code that we need to put in production before new releases have been merged upstream. It is used in combination with the normal PyPI index. I've started using local version identifiers for these custom releases, so that I can be sure that they won't conflict with upstream releases. Nobody but us should see these releases.
Unfortunately, just the act of uploading a new release with a local version identifier is breaking my build, because, it appears, it is using the new uploaded version, even though the lock file has been locked to the PyPI version (without the local version identifier). This gives me no opportunity to test such a local release version, which rather defeats the purpose of having a lockfile.
Expected result
I expect that once I've locked a set of packages for a set of package indexes, that it will only use the exact version as locked in the lockfile, and not automatically use local versions.
Actual result
It uses the local version (with the
+aspireduversion string appended), and then subsequently fails, appropriately, because the version it's trying to install doesn't have the same hash as the one that it was locked with.Steps to replicate
In order to independently replicate, you'll need to have access to an index that you can upload to. First create the pipfile and lock it before you upload a local version. If you don't have access to a private repository, you can use this repository, though you'll have to trust that I did things the way I'm saying.
https://github.com/ryanhiebert/pipenv-local
You need to have both indexes added to the file before you lock, so that you'll have the custom index in place when you go to install, and it sees that there's a new local version in the custom index.
Then upload a new version to the custom index, with the same version but with a local segment added (
3.0.1+aspiredu1in my example). Attempt to install from the lockfile again usingpipenv install, unchanged, and it will pull in from the custom index, instead of the locked PyPI version, causing the hash to mismatch and the installation to fail.$ pipenv --support
Pipenv version:
'2018.11.14'Pipenv location:
'/home/ryan/.local/lib/python3.6/site-packages/pipenv'Python location:
'/home/ryan/.pyenv/versions/3.6.6/bin/python3.6'Python installations found:
3.7.1:/home/ryan/.pyenv/versions/3.7.1/bin/python33.7.1:/home/ryan/.pyenv/versions/3.7.1/bin/python3.7m3.7.0:/home/ryan/.pyenv/versions/3.7.0/bin/python33.7.0:/home/ryan/.pyenv/versions/3.7.0/bin/python3.7m3.6.7:/home/ryan/.pyenv/versions/3.6.7/bin/python33.6.7:/home/ryan/.pyenv/versions/3.6.7/bin/python3.6m3.6.6:/home/ryan/.pyenv/versions/3.6.6/bin/python33.6.6:/home/ryan/.pyenv/versions/3.6.6/bin/python3.6m3.6.6:/usr/bin/python33.6.6:/usr/bin/python3.6m2.7.15rc1:/usr/bin/python2.7PEP 508 Information:
System environment variables:
CLUTTER_IM_MODULELS_COLORSLESSCLOSEXDG_MENU_PREFIXLANGGDM_LANGDISPLAYQT_STYLE_OVERRIDECOLORTERMPYENV_VIRTUALENV_INITXDG_VTNRSSH_AUTH_SOCKMANDATORY_PATHXDG_SESSION_IDXDG_GREETER_DATA_DIRUSERDESKTOP_SESSIONQT4_IM_MODULETEXTDOMAINDIRGNOME_TERMINAL_SCREENDEFAULTS_PATHQT_QPA_PLATFORMTHEMEPWDHOMETEXTDOMAINSSH_AGENT_PIDQT_ACCESSIBILITYXDG_SESSION_TYPEXDG_DATA_DIRSXDG_SESSION_DESKTOPGTK_MODULESTERMSHELLVTE_VERSIONXDG_SEAT_PATHQT_IM_MODULEXMODIFIERSIM_CONFIG_PHASEXDG_CURRENT_DESKTOPGPG_AGENT_INFOGNOME_TERMINAL_SERVICEXDG_SEATSHLVLPYENV_SHELLLANGUAGEGDMSESSIONGNOME_DESKTOP_SESSION_IDLOGNAMEDBUS_SESSION_BUS_ADDRESSXDG_RUNTIME_DIRXAUTHORITYXDG_SESSION_PATHXDG_CONFIG_DIRSPATHSESSION_MANAGERLESSOPENGTK_IM_MODULEOLDPWD_PIP_DISABLE_PIP_VERSION_CHECKPYTHONDONTWRITEBYTECODEPIP_SHIMS_BASE_MODULEPIP_PYTHON_PATHPYTHONFINDER_IGNORE_UNSUPPORTEDPipenv–specific environment variables:
Debug–specific environment variables:
PATH:/home/ryan/bin:/home/ryan/.local/bin:/home/ryan/.pyenv/plugins/pyenv-virtualenv/shims:/home/ryan/.pyenv/shims:/home/ryan/.pyenv/bin:/home/ryan/.local/bin:/home/ryan/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/binSHELL:/bin/bashLANG:en_US.UTF-8PWD:/home/ryan/Code/aspiredu/pipenv-localContents of
Pipfile('/home/ryan/Code/aspiredu/pipenv-local/Pipfile'):Contents of
Pipfile.lock('/home/ryan/Code/aspiredu/pipenv-local/Pipfile.lock'):{ "_meta": { "hash": { "sha256": "a1162dba4d383f6f7924058142a87bc04355d6e68529b0c51d14743a32316fb6" }, "pipfile-spec": 6, "requires": { "python_version": "3.7" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true }, { "name": "aspiredu", "url": "https://pypi.fury.io/OHXIw-9Umgf3jlbpG5ULSp97w456Q/aspiredu/", "verify_ssl": true } ] }, "default": { "django-cleanup": { "hashes": [ "sha256:53316b04b9c91725fce9eac252cdc762a988b5a1640aeee8ab43048ccf9d98bd" ], "index": "pypi", "version": "==3.0.1" } }, "develop": {} }The text was updated successfully, but these errors were encountered: