pipenv lock breaks when pip download is set to no-binary :all: in pip.conf

[simonpercivall]

If pipenv is used in an environment where the pip.conf download section contains no-binary = :all:, pipenv install will fail, because the has of the source archives doesn't match hash of the wheels.

This may be something fixable, or perhaps just something that would be helpful to document.

[sunshyland]

adb shell getprop > sulog.txt
adb shell set >> sulog.txt
adb shell toolbox id >> sulog.txt
adb shell busybox >> sulog.txt
adb shell su -v >> sulog.txt
adb shell su -V >> sulog.txt
adb shell su -h >> sulog.txt
adb shell su --self-test >> sulog.txt
adb shell ls -l /system/xbin/su >> sulog.txt
adb shell ls -lZ /system/xbin/su >> sulog.txt
adb shell ls -lZ /system/bin/toolbox >> sulog.txt
adb shell ls -l /system/bin >> sulog.txt
adb shell ls -l /system/xbin >> sulog.txt
adb shell ls -laR /su >> sulog.txt
adb shell ps >> sulog.txt

[nateprewitt]

So the issue here is that the Pipfile.lock has either been moved to a new environment, or the pip.conf file has been updated since the lock was generated. You can address this by generating a new Pipfile.lock with pipenv lock which will then use hashes from the wheels your system downloads. This does seem to somewhat defeat the purpose of the hashes, but it would be a bit absurd to try generating lock files for every possible system you might move a project to.

This issue is also present in Python 2 vs Python 3, in that different files are downloaded, so the hashes won't match.

I'm not immediately sure if there's a better way to address this than documenting it. A given Pipfile currently needs to generate it's lock file with the same system configuration it will be run on. @kennethreitz do you have any thoughts on this?

[simonpercivall]

The issue here was specifically that the hashes for pipenv lock were generated for the source archives, and pip install checked against wheels. But what you say is true as well.

Hashes for source dists would be portable, but would require the whole pipenv chain of commands to never use pre-built wheels.

[nateprewitt]

@simonpercivall, to be clear, are you saying a system with no-binary = :all: generated a lock file using source archives? That isn't something I've been able to reproduce, which is why I said the Pipfile.lock has either been moved to a new system, or the pip.conf was updated after the Pipfile.lock was created.

pipenv should play nice with either paradigm (archives vs wheels), you just have to use it consistently right now. If you run pipenv lock on a default pip setup, it will use archives which are more universal. If it's a modified pip environment, you need to make sure the lock file is only used on systems with those config settings present.

[simonpercivall]

Hmm, very weird that we have different experiences here. Are you on Linux? I know that pip (except for multilinux-wheels) downloads and installs from source archives there. For macOS, and for multilinux-wheels, pip defaults to downloading and installing pre-build wheels, if they are uploaded to PyPI.

So my experience would be that a default setup does not generate portable Pipfile.lock files. This, for instance, is true for a clean checkout of the pipenv repository. If I run pipenv install --two on macOS, it will not work, since the hashes are generated for source archives, and pip chooses uploaded wheels:

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    crayons==0.1.2 from https://pypi.python.org/packages/fb/96/5f68c8572c5af02b12c5244dd511e7120aeb523ce03285ffa29a4b3cda39/crayons-0.1.2-py2-none-any.whl#md5=af7aa9a4b3f365ed0b6ba853b0c8f2c9 (from -r /var/folders/5l/rj9yryv53h50m1zwnx30qp9h0000gn/T/tmpiwzhwx87-requirements.txt (line 1)):
        Expected sha256 5e17691605e564d63482067eb6327d01a584bbaf870beffd4456a3391bd8809d
             Got        6f51241d0c4faec1c04c1c0ac6a68f1d66a4655476ce1570b3f37e5166a599cc

[nateprewitt]

I've tried several configurations on macOS and can't generate what you're seeing. Can you please provide the following for me?

• python --version
• pipenv run python --version
• pipenv --version
• cat | head -1 .venv/bin/pip
• The output of the file at $PIP_CONFIG_FILE
• your Pipfile

[simonpercivall]

Hmm, I don't know if I'm just confused about what you're saying; perhaps we're actually agreeing.

Note that the pipenv repo at commit c54d661 switched from using the hash from crayons 0.1.2.tar.gz to using the hash from crayons-0.1.2-py2-none-any.whl, so retrying my example from the comment above will not work, since pip download downloads the wheel … that is, unless you set no-binary :all: in pip.conf.

Let's take this as example:

1. A clean checkout of the pipenv repository, at 1410b8b
2. Pipfile.lock contains (for instance):

"crayons": {
            "version": "==0.1.2",
            "hash": "sha256:6f51241d0c4faec1c04c1c0ac6a68f1d66a4655476ce1570b3f37e5166a599cc"
        },

this corresponds to the Python 2 whl. A pipenv install will work for Python 2, but not for Python 3, as you said, since Py3 uses different wheels. This is exactly as you wrote in your first comment … but then you said that pipenv lock uses the sdist archives on a default pip setup?

If I create that Pipfile.lock myself:

> python --version
Python 2.7.13
> pipenv --version
pipenv, version 3.2.13
> pip --version # if this matters
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)

1. Comment out everything in ~/.pip/pip.conf and ~/.pydistutils.cfg

> git remote get-url origin
git@github.com:kennethreitz/pipenv.git
> git rev-parse --short HEAD
1410b8b

> rm Pipfile.lock
> pipenv lock
> git diff Pipfile.lock | wc -l
       0
> pipenv run python --version
[...]
Python 2.7.13
> cat | head -1 .venv/bin/pip
#!/Volumes/Disk/code/pipenv.git/.venv/bin/python2.7

I.e. pipenv lock by default uses whatever pip download downloads, not the source archives.

4. Then, in ~/.pip/pip.conf enter:

[download]
no-binary = :all:

> rm Pipfile.lock
> pipenv lock
> git diff Pipfile.lock|wc -l
     174

With no-binary :all: in ~/.pip/pip.conf, we get the hashes from the source archives instead of the wheel.

[nateprewitt]

Yep, I think this seems to be a case of a few crossed wires (mostly mine) :) I now see what you were saying. My tests were showing source archives but I apparently chose packages that didn't have wheels (e.g. toml).

You're correct that pip prefers wheels and will only use source archives if there's no wheel available. So I think we're back to the original conflict of Python 2 hashes vs Python 3.

If you generate a Pipfile.lock in Python 2, you're likely going to get hashes with py2 wheels. It's possible, but not guaranteed, that these will be py2.py3 wheels which work cross version. Without guaranteed universal wheels, we can only support Python 2 with Pipfile.locks generated in Python 2. Similarly, we can only support Python 3 with Pipfile.locks generated in Python 3.

For the case of no-binary :all:, this should work cross version, but you'll need to create your lock file with this config in place. Anything generated with a default pipenv environment may not be compatible.

We can look into documenting this behaviour. That way users that want cross version compatibility at the expense of wheels should be able to set up their pip.conf to get it.

[simonpercivall]

Yes. Another alternative could be for pipenv to always use source archives, both for lock and install, since the --no-binary=:all: option is available for both pip install and pip download. I guess it's be a question of which solution is least bad for the most people.

Not using wheels makes it fully portable (and an option for pipenv could disable hashes+no-binary for just the install command, for people preferring speed over verifiability); using wheels makes installs faster and more reliable (I'm thinking of stuff like pandas, numpy and matplotlib here), but also makes it non-portable, both over architectures and over Python versions, which makes checking in and distributing Pipfile.lock-files actively harmful.

[nateprewitt]

@simonpercivall yeah, this is a question of better defaults. We could easily add --no-binary=:all: to out commands and my original write up was going to suggest that, but that completely eliminates wheels.

Using pip.conf at least makes this toggleable, whereas hardcoding --no-binary can't be changed without adding another flag. I'm not sure if @kennethreitz wants to keep adding flags right now, but perhaps we can take another look at the default here.

[kennethreitz]

i think the current implementation is fine for now, wheels are excellent, and should be used when available.

[nateprewitt]

There's the word then :) I'll write up configuring a pip.conf entry @simonpercivall and I think we'll leave it there for now.

[nateprewitt]

Also thanks for your time researching this, @simonpercivall!

[simonpercivall]

Thanks for the discussion.

Hopefully this situation will improve over time; for instance if pipenv lock stored multiple hashes, by arch/version, and if hashes for wheels on PyPI were downloadable without downloading the actual files.