Locked info for local wheel should contain hash of the wheel file itself

[uranusjr]

From #2390. Take googleappenginepipeline for example, here’s the lock file

{
    "_meta": {
        "hash": {
            "sha256": "6929c0ba32c595c656469787bbbe9ae6f7f840b5bc03b9e97dd518c3468f3672"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "2.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "googleappenginecloudstorageclient": {
            "hashes": [
                "sha256:b00ad9e7ca5a5524f07b6099b4c1e920cca9badc8e0d57cb6c1e2fb976356c95"
            ],
            "version": "==1.9.22.1"
        },
        "googleappenginepipeline": {
            "hashes": [
                "sha256:cef7f04172870db661c8ae26fc1417b1bb30bdcd0019b1cdd15c5c4cd3c0ae1c"
            ],
            "path": "./GoogleAppEnginePipeline-1.9.22.1-py2-none-any.whl",
            "version": "==1.9.22.1"
        }
    },
    "develop": {}
}

The hash cef7f04172870db661c8ae26fc1417b1bb30bdcd0019b1cdd15c5c4cd3c0ae1c is the source archive’s hash from PyPI, but that does not match the wheel’s, and causes an failure when you run pipenv sync:

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    GoogleAppEnginePipeline==1.9.22.1 from file:///.../GoogleAppEnginePipeline-1.9.22.1-py2-none-any.whl (from -r ... (line 1)):
        Expected sha256 cef7f04172870db661c8ae26fc1417b1bb30bdcd0019b1cdd15c5c4cd3c0ae1c
             Got        13210232486bf683d928aae1a2d9d97edada6d2d83063dc23b71fb6bbcee9874

The solution to this, I think, is to include the wheel file’s hash with those from PyPI. The user should be responsible for providing a legit wheel file during locking, and if (s)he does, this hash can be used to ensure the wheel file used during sync is also legit.

[Laski]

Any workaround until this is fixed? Using --skip-lock every time kinda defeats the purpose of pipenv.

[Laski]

Found a workaround: changing the name of the package to a random string (or any string that's not a valid pypi package) prevents pipenv from searching the hash in pypi, but the installation works because a local wheel does not seem to need a correct package name.

This fails:

mysqlclient = {path = "./dependencies/mysqlclient-1.3.13-cp35-cp35m-win32.whl", os_name = "== 'nt'"}

But this works:

mysqlclient-win = {path = "./dependencies/mysqlclient-1.3.13-cp35-cp35m-win32.whl", os_name = "== 'nt'"}

And this should too (haven't tested it tho):

majhxhxoic = {path = "./dependencies/mysqlclient-1.3.13-cp35-cp35m-win32.whl", os_name = "== 'nt'"}

[andyneff]

This is also an issue for downloaded wheels that are not from pypi, and thus should not be compared to it. pipenv attempts to compare to hashes on pypi which just doesn't make sense, nor is it security, since pipenv install --keep-outdated doesn't have this issue, and actually works.

Also seen in #3053 #2863 and #2757

TLDR;

If the wheel is local/an external file, not a pypi file, the hash should represent that file or not be there at all, but should not be the hash of pypi

Works

pipenv install https://github.com/uqfoundation/dill/archive/dill-0.2.8.zip
pipenv --rm
pipenv install --keep-outdated

Just showing an example of setting up an environment, pretending I'm "starting over", and then "I want to reinstall what's in the Pipfile/Pipfile.lock". But in this case, the zip file is left alone, in peace, like you would expect. pipenv install also works, as does pipenv lock

Also works as expected

pipenv install https://github.com/uqfoundation/dill/archive/dill-0.2.8.zip#egg=dill
...

Does not work as intended

pipenv install http://download.pytorch.org/whl/cu92/torch-0.4.1-cp36-cp36m-linux_x86_64.whl

And I can't even get past that, without using @Laski 's workaround, which I can't figure out from the pipenv cli.

Other incomplete workarounds

#Egg

pipenv install http://download.pytorch.org/whl/cu92/torch-0.4.1-cp36-cp36m-linux_x86_64.whl#egg=foobar
pipenv --rm
pipenv install --keep-outdated

Will not actually work. The Pipfile will contain a torch entry (not foobar) and the Pipfile,lock will contain none. So pipenv install --keep-outdated does nothing. pipenv install or pipenv lock just repeats the same hash issue.

Editing Pipfile.lock

You can edit your Pipfile.lock file to look like this

{
    "_meta": {
        "hash": {
            "sha256": "a498ccb6a730b630bed3d7ce40bdb058e9f64cd962f70bc640366b5f9d2d3c38"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "torch": {
            "file": "http://download.pytorch.org/whl/cu92/torch-0.4.1-cp36-cp36m-linux_x86_64.whl"
        }
    },
    "develop": {}
}

And now, pipenv install --keep-outdated works, but any attempt to pipenv lock will lead to the same hash disaster. So this is not a long term solution, as a project using this workaround can no longer breath.

Pytorch background

Pytorch links with a library (called cuda) that can come in a number of incompatible versions, 8.0, 9.0, 9.1, etc... So they decided to put the 9.0 version on pypi and you have to download the wheel for any other version. While on the surface it may seem weird, this is just a use case of "using a wheel not on pypi". And currently, I can't find a work around short of manually editing the Pipfile, to make this work. The current behavior seems to be buggy, and does not add "security", as --keep-outdated makes this work partially.

[ed-alertedh]

I can confirm @Laski 's workaround is the easiest way to deal with this for now.