Pipfile.lock contains md5 hashes on my docker build #4516

eblis · 2020-11-04T15:12:56Z

Issue description

I have a problem with pipfile.lock and I can't find out what is causing it.
On my local machine, which is a Windows 10 if I remove Pipfile.lock and issue a pipenv install I get sha256 hashes in Pipfile.lock, as seen here:

        "coloredlogs": {
            "hashes": [
                "sha256:14525820de1f749f836763a6631a48d4854f59160c42d4e81a3ecb625514615c"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
            "version": "==14.101"
        },

However, on my build machine, which is a Linux server building a docker container with Python 3.8.6 I get the following error when running pipenv install

Updated Pipfile.lock (6c5e70)!
Installing dependencies from Pipfile.lock (6c5e70)...
An error occurred while installing coloredlogs==14.101; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' --hash=md5:74c9ee3424df05e34f093b130e0fea9f! Will try again.
An error occurred while installing petpcomms==0.1.12 --hash=md5:04d97ee07edaaacab3cd72268b0aebae! Will try again.
An error occurred while installing petpmvcs==0.1.50 --hash=md5:5e111dbd663627bf0586c76024362d65! Will try again.
Installing initially failed dependencies...
[InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/cli/command.py", line 232, in install
[InstallError]:       retcode = do_install(
[InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 2051, in do_install
[InstallError]:       do_init(
[InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 1304, in do_init
[InstallError]:       do_install_dependencies(
[InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 899, in do_install_dependencies
[InstallError]:       batch_install(
[InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 796, in batch_install
[InstallError]:       _cleanup_procs(procs, failed_deps_queue, retry=retry)
[InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 703, in _cleanup_procs
[InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: Usage: pip [options]
[pipenv.exceptions.InstallError]: 
[pipenv.exceptions.InstallError]: ERROR: Invalid requirement: coloredlogs==14.101; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' --hash=md5:74c9ee3424df05e34f093b130e0fea9f
[pipenv.exceptions.InstallError]: pip: error: Allowed hash algorithms for --hash are sha256, sha384, sha512.
ERROR: Couldn't install package: coloredlogs

Expected result

Hashes should be created in sha256 format on the Linux Docker Python image, similar to the way they're created on Windows.

Actual result

It looks like the Pipfile.lock uses md5 hashes in my docker container.

Steps to replicate

Use a docker container with Python 3.8.6
Install latest pip (already up to date for now)
Install latest pipenv

$ pipenv --support

Pipenv version: '2020.11.4'
Pipenv location: '/usr/local/lib/python3.8/site-packages/pipenv'
Python location: '/usr/local/bin/python'
Python installations found:

3.8.6: /usr/local/bin/python3
3.8.6: /usr/local/bin/python3.8
3.7.3: /usr/bin/python3
3.7.3: /usr/bin/python3.7
3.7.3: /usr/bin/python3.7m
2.7.16: /usr/bin/python2
2.7.16: /usr/bin/python2.7
PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.12.14-122.32-default',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Aug 5 12:59:08 UTC 2020 (477c426)',
 'python_full_version': '3.8.6',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

HTTPS_PROXY
HOSTNAME
PYTHON_PIP_VERSION
PETP_ENVIRONMENT
HOME
GPG_KEY
PYTHON_GET_PIP_URL
PATH
LANG
PYTHON_VERSION
PWD
PYTHON_GET_PIP_SHA256
HTTP_PROXY
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED
Pipenv–specific environment variables:
Debug–specific environment variables:
PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG: C.UTF-8
PWD: /usr/src/app

Contents of Pipfile ('/usr/src/app/Pipfile'):

[[source]]
name = "Artifactory"
url = "https://highway.porsche.com/artifactory/api/pypi/pypi-all/simple"
verify_ssl = true
[dev-packages]
pytest = "*"
pytest-bdd = "*"
pytest-mock = "*"
prospector = ">=1.3.0.dev4"
pytest-cov = "*"
[packages]
alembic = "*"
cachetools = "*"
coverage = "*"
dataclasses-json = "*"
ldap3 = "*"
petpcomms = "*"
petpmvcs = "*"
pinject = "*"
pyaml = "*"
python-dateutil = "*"
pymongo = "*"
argon2-cffi = "*"
[requires]
python_version = "3.8"

The text was updated successfully, but these errors were encountered:

deltamualpha · 2020-11-04T21:30:16Z

Does this have something to do with #4513, I wonder?

frostming · 2020-11-05T01:04:41Z

Can you paste all the links(with full URL) listed in https://highway.porsche.com/artifactory/api/pypi/pypi-all/simple/coloredlogs ?

Does this have something to do with #4513, I wonder?

Nope, they are irrelevant

eblis · 2020-11-05T07:54:39Z

Artifactory links

I see that coloredlogs 14.0 has a sha256 URL: ../../packages/packages/84/1b/1ecdd371fa68839cfbda15cc671d0f6c92d2c42688df995a9bf6e36f3511/coloredlogs-14.0.tar.gz#sha256=a1fab193d2053aa6c0a97608c4342d031f1f93a3d1218432c59322441d31a505
but coloredlogs 14.101 has a md5 one ../../coloredlogs/14.101/coloredlogs-14.101.tar.gz#md5=74c9ee3424df05e34f093b130e0fea9f

frostming · 2020-11-05T08:06:21Z

Thanks for your information, I have submitted a fix for this

eblis · 2020-11-05T08:52:02Z

Thanks

joshsleeper · 2020-11-06T22:28:57Z

@frostming fyi even when installing pipenv from the latest source I'm still seeing md5:... hashes showing up in my Pipenv.lock files, so unless I'm misunderstanding it looks like #4519 didn't fix it completely.

I'm also using Artifactory internally for both hosting internal packages and as a simple caching proxy to PyPI.

a simple ripgrep search on the installed pipenv source confirms that the change from #4519 is present:

$ pipenv --version
pipenv, version 2020.11.4.dev0

$ rg "FAVORITE_HASH, new" ~/.local/pipx/venvs/pipenv
/home/jsleeper/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py
85:                hash_value = "{}:{}".format(FAVORITE_HASH, new_location.hash)

edit 1

so I did a little more testing by undoing all of my lockfile changes in that particular commit and re-running pipenv lock with the latest pipenv release (not from source)... and it worked just fine, no md5 hashes to be found anywhere. 😑

so I take that to mean there's some behavior there that's not working quite right, but now I'm very unclear on what or how. sorry that's probably super unhelpful

frostming · 2020-11-06T23:49:25Z

yes old hashes will remain in the lockfile unless you delete it and relock

joshsleeper · 2020-11-06T23:59:41Z

right, I guess was saying that in my case a lockfile without md5 hashes suddenly had them after running pipenv lock with the latest release, but I'm not finding the correct order of operations to replicate it again.

jvstein · 2020-11-12T19:26:32Z

I observed md5 hashes getting into my Pipfile.lock files on a number of packages coming from an Artifactory source, even after deleting the lock file. The behavior was consistent using older versions of pipenv, pipenv==2020-11-4, and the master branch, which included #4519.

Ultimately, the source was a number of md5 hashes cached in the ~/.cache/pipenv/hash-cache/ folder. Deleting them solved my problem and I'm now getting sha256 hashes in my Pipfile.lock.

grep -lr md5 ~/.cache/pipenv/hash-cache | xargs rm

byoungb · 2020-11-13T14:46:55Z

@jvstein thank you so much this has been driving me crazy since yesterday. Any idea where those md5s in the cache came from?

jvstein · 2020-11-14T03:42:06Z

@byoungb I'm honestly not sure. I wasn't able to reproduce the creation of those cached md5 hashes. I do suspect it's related to RTFACT-18495 because it was only affecting our custom packages, nothing from PyPI.

mnahkies · 2020-11-16T13:57:14Z

@jvstein thank you so much this has been driving me crazy since yesterday. Any idea where those md5s in the cache came from?

This has been driving me nuts as well, and clearing the cache fixed it.

As far as I can tell my previous attempts (older version of pipenv) to install the packages from artifactory had cached the hash, and then the updated version of pipenv with the fix was still using the cached value.

zaufi · 2020-11-16T15:35:08Z

I've also just hit this bug in Artifactory %(( I can't install a just-released package and after clearing the pip cache I cant install any package from our repos in Artifactory cloud.

Anyone has found a workaround?

eblis · 2020-11-16T15:37:23Z

You can get around it with --skip-lock

jaspalsingh94 · 2020-11-16T18:27:22Z

@frostming None of the above fixed the issue for me. For our internal packages hosted on gemfury, the lock file still has md5 hashes. I am using an ubuntu 18.05 docker image with python3.7.5.
pipenv version 2020.11.4.
pip version 20.2.4.
I deleted the lock file, rebuilt the image from scratch and recreated the lock file. The gemfury packages were populated with md5 hashes and all other packaged have sha256 hashes. Gemfury shows sha256 hashes are available for our packages.

My temporary fix was to go back to an older commit and manually copy the sha256 hashes into the current lock file. Re-locking the lock file would break this again though.

frostming · 2020-11-17T01:47:53Z

To clear the caches:

Update to the latest release, which is 2020.11.15 currently
Remove hash caches located at:
- ~/.cache/pipenv/hash-cache for Linux
- ~/Library/Cache/pipenv/hash-cache for macOS
- %USERPROFILE%\AppData\Local\pipenv\pipenv\Cache\hash-cache for Windows
And regenerate the lock file

Benhgift · 2020-11-18T18:51:45Z

@frostming I'm on OSX and followed your steps without success, and then again after deleting the whole cache folder, reinstalling pipenv, and deleting the lock file again.

It still puts the md5 hashes in the lock file.

Edit: I solved it by copying in an old sha256 hash from a previous commit into the lock file.

mckornfield · 2021-01-05T18:02:14Z

It might depend on what version of OSX you're on, I'm on Catalina, and for me, this allowed me to run pipenv lock and get the sha256 hash from an artifactory PyPi (although the md5 hash was also there...), similar to @jvstein 's answer. I also had to delete my Pipenv.lock file, otherwise it simply appended to the md5 one

grep -lr md5 ~/Library/Caches/pipenv/hash-cache | xargs rm

Note the path was ~/Library/Caches/pipenv/hash-cache not ~/Library/Cache/pipenv/hash-cache, at least on Catalina.

AmarendraSingh88 · 2021-06-23T19:05:02Z

Thanks @frostming, Clearing the cache was required but with the older version of pipenv, I still was not able to generate only the SHA hashes. I was on version 2020.5.28.
I have upgraded to the latest pipenv version 2021.5.29, cleared the cache, and regenerating the lock file did the trick. I now have only SHA hashes in Pipfile.lock and it is not complaining about the 'Allowed hash algo'

frostming mentioned this issue Nov 5, 2020

Restrict the acceptable hash type to SHA256 only #4519

Merged

frostming closed this as completed in #4519 Nov 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipfile.lock contains md5 hashes on my docker build #4516

Pipfile.lock contains md5 hashes on my docker build #4516

eblis commented Nov 4, 2020 •

edited by frostming

Loading

deltamualpha commented Nov 4, 2020

frostming commented Nov 5, 2020 •

edited

Loading

eblis commented Nov 5, 2020

frostming commented Nov 5, 2020

eblis commented Nov 5, 2020

joshsleeper commented Nov 6, 2020 •

edited

Loading

frostming commented Nov 6, 2020

joshsleeper commented Nov 6, 2020

jvstein commented Nov 12, 2020 •

edited

Loading

byoungb commented Nov 13, 2020

jvstein commented Nov 14, 2020

mnahkies commented Nov 16, 2020

zaufi commented Nov 16, 2020

eblis commented Nov 16, 2020

jaspalsingh94 commented Nov 16, 2020

frostming commented Nov 17, 2020 •

edited

Loading

Benhgift commented Nov 18, 2020 •

edited

Loading

mckornfield commented Jan 5, 2021

AmarendraSingh88 commented Jun 23, 2021

Pipfile.lock contains md5 hashes on my docker build #4516

Pipfile.lock contains md5 hashes on my docker build #4516

Comments

eblis commented Nov 4, 2020 • edited by frostming Loading

Issue description

Expected result

Actual result

Steps to replicate

deltamualpha commented Nov 4, 2020

frostming commented Nov 5, 2020 • edited Loading

eblis commented Nov 5, 2020

frostming commented Nov 5, 2020

eblis commented Nov 5, 2020

joshsleeper commented Nov 6, 2020 • edited Loading

edit 1

frostming commented Nov 6, 2020

joshsleeper commented Nov 6, 2020

jvstein commented Nov 12, 2020 • edited Loading

byoungb commented Nov 13, 2020

jvstein commented Nov 14, 2020

mnahkies commented Nov 16, 2020

zaufi commented Nov 16, 2020

eblis commented Nov 16, 2020

jaspalsingh94 commented Nov 16, 2020

frostming commented Nov 17, 2020 • edited Loading

Benhgift commented Nov 18, 2020 • edited Loading

mckornfield commented Jan 5, 2021

AmarendraSingh88 commented Jun 23, 2021

eblis commented Nov 4, 2020 •

edited by frostming

Loading

frostming commented Nov 5, 2020 •

edited

Loading

joshsleeper commented Nov 6, 2020 •

edited

Loading

jvstein commented Nov 12, 2020 •

edited

Loading

frostming commented Nov 17, 2020 •

edited

Loading

Benhgift commented Nov 18, 2020 •

edited

Loading