Pipenv lock still takes way too long to complete

[mjpieters]

pipenv lock is still taking up to 30 minutes to complete for my medium-sized project.

While #4403 was closed as fixed and indeed no full tree copies are made any more, pipenv lock is still unworkably slow. What used to take 50 seconds under 2018.11.26, now takes 30 minutes to figure out that nothing changed:

$ export PIPENV_INSTALL_TIMEOUT=10000
$ cp Pipfile.lock Pipfile.lock.before
$ time pipenv lock
Locking [dev-packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (0e1198)!

real	29m19.701s
user	28m39.300s
sys	4m8.222s
$ diff -su Pipfile.lock.old Pipfile.lock
Files Pipfile.lock.old and Pipfile.lock are identical

This is making it untenable to keep using Pipenv.

Full pipenv lock -v output in a gist.

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
flake8 = "*"
flake8-bugbear = "*"
black = "==19.10b0"
pre-commit = "*"
pytest = "*"
pytest-cov = "*"

[packages]
project-name = {editable = true,path = "."}
"zc.buildout" = "*"

[requires]
python_version = "3.6"

where project-name (redacted name) lists the following install-requires lines:

install_requires =
    apache-airflow[aws,celery,postgres,redis] >= 1.10.11
    airflow_multi_dagrun
    airflow-prometheus-exporter

---

$ pipenv --support

Pipenv version: '2020.8.13'

Pipenv location: '/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2020.8.13/libexec/bin/python3.8'

Python installations found:

• 3.9.0: /Users/mj/bin/python3.9
• 3.8.5: /usr/local/bin/python3
• 3.8.5: /usr/local/bin/python3.8
• 3.8.3: /Users/mj/bin/python3.8
• 3.7.7: /Users/mj/bin/python3.7
• 3.7.6: /opt/miniconda3/bin/python3
• 3.7.6: /opt/miniconda3/bin/python3.7m
• 3.7.6: /opt/miniconda3/bin/python3.7
• 3.7.3: /Users/mj/.pyenv/versions/3.7.3/bin/python3
• 3.6.10: /Users/mj/bin/python3.6
• 3.6.1: /Users/mj/bin/pypy3
• 3.5.9: /Users/mj/bin/python3.5
• 3.4.10: /Users/mj/bin/python3.4
• 3.3.7: /Users/mj/bin/python3.3
• 3.2.6: /Users/mj/bin/python3.2
• 2.7.18: /Users/mj/bin/python2.7
• 2.7.16: /usr/bin/python2.7
• 2.6.9: /Users/mj/bin/python2.6
• 2.5.6: /Users/mj/bin/python2.5
• 2.4.6: /Users/mj/bin/python2.4

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Wed May 27 17:00:02 PDT '
                     '2020; root:xnu-4570.71.80.1~1/RELEASE_X86_64',
 'python_full_version': '3.8.5',
 'python_version': '3.8',
 'sys_platform': 'darwin'}

System environment variables:

• PATH
• AUTOJUMP_ERROR_PATH
• NVM_INC
• TERM_PROGRAM
• GIT_PROMPT_END
• rvm_bin_path
• AUTOJUMP_SOURCED
• GEM_HOME
• NVM_CD_FLAGS
• SHELL
• TERM
• HISTSIZE
• TMPDIR
• COPYFILE_DISABLE
• IRBRC
• Apple_PubSub_Socket_Render
• CONDA_SHLVL
• PERL5LIB
• TERM_PROGRAM_VERSION
• CONDA_PROMPT_MODIFIER
• TERM_SESSION_ID
• MY_RUBY_HOME
• PERL_MB_OPT
• LC_ALL
• USER
• HISTFILESIZE
• NVM_DIR
• COMMAND_MODE
• CONDA_EXE
• rvm_path
• SSH_AUTH_SOCK
• __CF_USER_TEXT_ENCODING
• PTPIMG_API_KEY
• _CE_CONDA
• rvm_prefix
• _
• CONDA_PREFIX
• PWD
• HOMEBREW_GITHUB_API_TOKEN
• EDITOR
• LANG
• ITERM_PROFILE
• PYTHONSTARTUP
• XPC_FLAGS
• ITERM_ORIG_PS1
• XPC_SERVICE_NAME
• HISTCONTROL
• _CE_M
• FIGNORE
• rvm_version
• COLORFGBG
• HOME
• SHLVL
• LC_TERMINAL_VERSION
• ITERM_PREV_PS1
• ITERM_SESSION_ID
• PERL_LOCAL_LIB_ROOT
• LOGNAME
• CONDA_PYTHON_EXE
• LC_CTYPE
• GEM_PATH
• GIT_PROMPT_ONLY_IN_REPO
• CONDA_DEFAULT_ENV
• NVM_BIN
• DISPLAY
• LC_TERMINAL
• HGSRC
• SECURITYSESSIONID
• RUBY_VERSION
• PERL_MM_OPT
• COLORTERM
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/Cellar/pipenv/2020.8.13/libexec/tools:/usr/local/google-sdks/google-cloud-sdk/bin:/Users/mj/perl5/bin:/Users/mj/.nvm/versions/node/v12.14.1/bin:/Users/mj/.rvm/gems/ruby-2.3.0/bin:/Users/mj/.rvm/gems/ruby-2.3.0@global/bin:/Users/mj/.rvm/rubies/ruby-2.3.0/bin:/opt/miniconda3/bin:/opt/miniconda3/condabin:/Users/mj/bin:/Users/mj/Development/Library/zopatista_setup/bin:/Users/mj/Development/Library/zopatista_setup/audio:/usr/local/Cellar/ruby/2.7.1_2/bin:/Users/mj/.cargo/bin:/Users/mj/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/usr/local/sbin:/opt/X11/bin:/Applications/Postgres.app/Contents/Versions/latest/bin:/Users/mj/.local/bin:/Users/mj/.rvm/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin
• SHELL: /bin/bash
• EDITOR: vim
• LANG: en_GB.UTF-8
• PWD: /Users/mj

---

[conradg]

Thanks for raising this @mjpieters, thought I was going crazy. Downgrading to 2018.11.26 speeds things up massively for me too

[witsch]

Thanks for raising this @mjpieters, thought I was going crazy. Downgrading to 2018.11.26 speeds things up massively for me too

Yes, +1! Thanks for looking into this, Martijn! We had exactly the same issue (lots of big tmp directories), but sorta gave up for the time being… I couldn't really argue much when pipenv lock took 10+ minutes for simply adding another package.

(and hi, btw 👋)

[mgmarino]

A "workaround" that may work (and what we have used from time to time) is to remove the lock file and rerun the lock. This may also hint at the underlying problem, which may have something to do with how pipenv is using the existing lock file during the lock process.

[Can-Sahin]

Deleting the Pipfile.lock speeds things up for sure. I just install all the packages from scratch and run pipenv lock without changing anything and it gets stuck eventhough there is no change at all

[frostming]

Can you have a try on the master branch? I recently disabled the copying of the local source tree.

[mjpieters]

Can you have a try on the master branch? I recently disabled the copying of the local source tree.

Do you mean #4421? That's been part of the 2020.8.13 release that I tested with for this issue. I did note that the tree copying no longer takes place.

I also don't see any changes on master since 2020.8.13 was released that look relevant. Did I miss something?

[frostming]

If I remember correctly, 2018.11.26 didn't resolve transient dependencies of an editable package. You can compare the lock files generated by both versions.

[mjpieters]

If I remember correctly, 2018.11.26 didn't resolve transient dependencies of an editable package. You can compare the lock files generated by both versions.

I see no differences that look problematic. With 2020.8.13.dev0 (master branch) I see markers entries added. The dependencies I see added are ones that were due to other dependencies having been upgraded (I deleted the lock file to see if there was much speed difference).

With the lock file deleted and using 2020.8.13.dev0, a pipenv lock took 10 minutes.

[frostming]

I removed all other dependencies except the editable one. It turns out no significant difference is seen in the time costs of 2018.11.26 and 2020.8.13. It will be great if you can narrow down which package brings the performance degrading.

I created a demo repo to showcase the issue, which you can clone and adjust to reproduce the performance issue:
https://github.com/frostming/pipenv-lock-performance

$ docker build . --build-arg PIPENV_VER=2020.8.13 -t pipenv-bug
$ docker build . --build-arg PIPENV_VER=2018.11.26 -t pipenv-bug

[mjpieters]

Sorry, I meant to get back to this but current project work is keeping me over-utilised. I do still find Pipenv to be excruciatingly slow, and regularly have to delete the lock file just to get a solution, but that's not really very actionable feedback.

[alexandrerays]

I'm having this problem in February 2021. Even deleting lock file, it is still very slow.

[tombohub]

Same here, deleting lock file doesn't help

[GiuseppeLaurenza]

same here, there are any news?

[Eortvald]

same here

[Real-Gecko]

Version 2020.11.15, does not take half an hour for me, but couple of minutes is too much for creating 40 KiB JSON file.

[chialunwu]

Why pipenv why so slow? Why?

[Nytelife26]

This issue is still present in April of 2021. I have yet to look into why this is, but I know it doesn't have to be this way because there are other managers that work better and faster.

For instance, I recommend using Poetry until this is fixed.
I know it seems counter-intuitive and disrespectful to recommend another project, but as of the present moment, the speed of pipenv makes it unusable, and I'd rather put an alternative on the table than leave people to wait.

[Real-Gecko]

I did not investigate the code, but I think it happens because on every run Pipenv checks for updates even though no one asks for this, thus it makes a lot of queries, calculates a lot of hashes for Pipfile.lock etc. So the bigger your .venv is the worse things get. For example right now I installed one package, but instead of just installing it and it's dependency Pipenv also updated three packages. As I already had 101 packages installed you can imagine the process was quite time consuming.

[CharString]

On 22 April 2021 02:53:29 CEST, Tyler J Russell ***@***.***> wrote: This issue is still present in April of 2021. I have yet to look into why this is, but I know it doesn't *have* to be this way because there are other managers that work better and faster. For instance, I recommend using Poetry until this is fixed.

Or have a look at pdm. You can make it use venvs instead of __pypackages__, if you want. And it can import your Pipfile.

[Nytelife26]

Or have a look at pdm. You can make it use venvs instead of __pypackages__, if you want. And it can import your Pipfile.

Interesting. I'll take a look later. I like Poetry, though, because it addresses another issue I noticed with pipenv - it uses a standard from PEP 518 that's likely to work with other toolchains (`pyproject.toml` as opposed to `Pipfile`).

[Nytelife26]

Interestingly enough, that's actually the same with package managers from other ecosystems, npm, yarn, etc, all update the lockfile when you install the project unless you specify otherwise. They shouldn't, but it proves a point - it doesn't have to be this painfully slow at all.

[Real-Gecko]

The biggest problem IMO is this spinner:

It's not really informative, one cannot tell exactly what's exactly going on right now. What if I have network problems and Pipenv experiences request timeouts? I wouldn't know, because all progress is hidden behind spinner.

[felix-ht]

just move to poetry - faster locks overall better performance

[Nytelife26]

just move to poetry

That's not really constructive at all. It's better to work on helping people here. I only suggested it as a temporary solution so people don't have to deal with the hassle while they wait for a fix.

You wouldn't walk into Google's headquarters and tell everyone to "just use DuckDuckGo", would you? It's basic respect.

[Nytelife26]

I'll be running tests later. I think the best way to resolve this is to time each stage of the lock and see which ones take the longest so we can determine where the problem comes from and attempt to fix it.

[mgmarino]

pipenv spawns a number of subprocesses during the lock, does this take any of that into account?

** edit ** I have found this is also why a lot of the output is hidden during verbose running; the stdout/stderr of the worker processes are not propagated up.

[Nytelife26]

pipenv spawns a number of subprocesses during the lock, does this take any of that into account?

It'll time exactly as long as each step takes to complete. Does it run the `do_lock` in worker threads, or spawn the worker threads from `do_lock`? I couldn't tell when I looked.

[Real-Gecko]

I've found the way to profile pipenv install, take a look at this repo for instructions. The result is *.calltree file which can be opened kcachegrind. I totally do not understand how to interpret this results, but I'm sure someone can figure it out.

[Nytelife26]

I've found the way to profile `pipenv install`

Marvellous. Thank you. I'll investigate later - although, at a glance, the presence of time.sleep is interesting.

[mjpieters]

I'll investigate later - although, at a glance, the presence of time.sleep is interesting.

I'm pretty sure that those are inside pexpect, used to monitor each pip process spawned. It just means that there is a lot of waiting around for those processes.

[mjpieters]

Yup, it's the vendored pexpect wait loop. expect_loop() function definition:

pipenv/pipenv/vendor/pexpect/expect.py

Lines 153 to 184 in 275f7e1

	def expect_loop(self, timeout=-1):
	"""Blocking expect"""
	spawn = self.spawn
	if timeout is not None:
	end_time = time.time() + timeout
	try:
	idx = self.existing_data()
	if idx is not None:
	return idx
	while True:
	# No match at this point
	if (timeout is not None) and (timeout < 0):
	return self.timeout()
	# Still have time left, so read more data
	incoming = spawn.read_nonblocking(spawn.maxread, timeout)
	if self.spawn.delayafterread is not None:
	time.sleep(self.spawn.delayafterread)
	idx = self.new_data(incoming)
	# Keep reading until exception or return.
	if idx is not None:
	return idx
	if timeout is not None:
	timeout = end_time - time.time()
	except EOF as e:
	return self.eof(e)
	except TIMEOUT as e:
	return self.timeout(e)
	except:
	self.errored()
	raise

and self.spawn.delayafterread is set to 0.0001 to release the GIL:

pipenv/pipenv/vendor/pexpect/spawnbase.py

Lines 72 to 77 in 275f7e1

	# Delay in seconds to sleep after each call to read_nonblocking().
	# Set this to None to skip the time.sleep() call completely: that
	# would restore the behavior from pexpect-2.0 (for performance
	# reasons or because you don't want to release Python's global
	# interpreter lock).
	self.delayafterread = 0.0001

Note that there's no point in setting it to None here, as it is not pexpect that is the problem. It's the pip subprocess, which hasn't been profiled.

[gshpychka]

Help me understand - is this acknowledged and are there plans to address this?

[gaetangr]

I have implemented Pipenv with my team recently, but the time it take to install even one package is crazy, I tried with panda, it took between 5mn to 8mn.

41 packages takes around 20mn to install.

I have tried the following :

• Deleting my virtuelenv
• Deleting Pipfile.lock

Nothing work so far

[Luca-Terrazzan]

Same problem for me; locking is so slow that we are currently within the very dangerous situation of avoiding to update the lockfile whenever possible to avoid losing hours of waiting times

[trallnag]

To all the people thinking that Poetry is the answer and is perfectly fast in all cases: You are wrong. Both Pipenv and Poetry are having these issues partly due to the nature how dependency management works / does not work in Python.

[Real-Gecko]

To all the people thinking that Poetry is the answer and is perfectly fast in all cases: You are wrong. Both Pipenv and Poetry are having these issues partly due to the nature how dependency management works / does not work in Python.

Totally agree, with lot's of dependencies it's slow too.

[khimaros]

anecdotally, switching my medium sized project to poetry solved all of my performance issues.

[FlavourDave92]

Same problem for me; locking is so slow that we are currently within the very dangerous situation of avoiding to update the lockfile whenever possible to avoid losing hours of waiting times

Same, this simply feels wrong but often locking just takes ages…

[sommelon]

I am using WSL on windows and it was taking literally half an hour or so to finish with locking

I'm also trying to use pipenv lock in the WSL and it takes much much longer compared to locking in Windows. Don't know if it's a problem with, WSL, ubuntu or pipenv or something else.

[tbrodbeck]

For some reason locking pandas takes soooo long in my project. I even activated verbose mode, but still it does not give me any information what it is doing for 10min now... (found another "closed" issue: #1198)

Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('rpi.gpio'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('rpi.gpio'), None)
Reporter.adding_requirement(SpecifierRequirement('pytest'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('pytest'), None)
Reporter.adding_requirement(SpecifierRequirement('pyyaml'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('pyyaml'), None)
Reporter.adding_requirement(SpecifierRequirement('pyinstaller'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('pyinstaller'), None)
Reporter.adding_requirement(SpecifierRequirement('pandas'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('pandas'), None)
⠼ Locking...

Is there a way how to solve this with pipenv?

[matteius]

@tbrodbeck What version are you on? Can you try the branch linked off this issue? #4926

[CharString]

@tbrodbeck I thinks it's bad taste to advocate another project here. So try the different branch. But there are alternatives to poetry, like pdm.

[tbrodbeck]

@matteius I am on version 2022.1.8 and Python 3.9.2

I tried to switch to the last branch mentioned, but I cannot access it:

fatal: invalid reference: issues-4651-infiite-iteration-take4

[matteius]

@tbrodbeck I am sorry for I should have been more clear that its this passing branch with the pip 22.0.4 vendoring changes: #4969

[tbrodbeck]

I think you would need to provide me some installation instructions. I was not able to install the branch with python3 setup.py install or sudo python3 setup.py install

[matteius]

@tbrodbeck Hmmm, I seem to recall having issues with the install command as well, so I have been installing it using python3 setup.py develop --user to install it at user level which is preferred, if you need it at a system level you can drop the --user flag.

[tbrodbeck]

Hmm I am still getting permission denied:

t@v2:~/pipenv $ python3 setup.py develop --user
running develop
/home/t/.local/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/t/.local/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running egg_info
error: [Errno 13] Permission denied

[matteius]

@tbrodbeck I'd rather not ping all these people with each update -- perhaps make a new issue report and we can discuss more. I suspect you need to pip uninstall pipenv first if you are getting a permission denied trying to install to --user location, like its already installed there.

[em1208]

Pipenv unfortunately it is terribly slow in my case. Locking is taking more than 30min and still has not completed.

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.7"

[packages]
Babel = ">=2.5"
bleach = "<3.0"
"boto3" = "<=1.9.183"
Django = ">=1.11,<=2.2.28"
dj_database_url = ">=0.3.0"
dj_email_url = ">=0.0.4"
django-babel = ">=0.6.1"
django-bootstrap4 = ">=0.0.4"
django-cache-url = ">=1.0.0"
django-countries = ">=5.2"
django-filter = ">=2.0"
django-graphql-jwt = ">=0.1.13,<=0.2.1"
pyjwt= "==1.7.1"
django-impersonate = ">=1.3"
django-measurement = "==3.0.0"
django-mptt = ">=0.7.1"
django-phonenumber-field = ">=1.3.0,<3.0"
django-prices = ">=1.0.2, <2.0"
django-prices-openexchangerates = ">=1.0.0"
django-prices-vatlayer = ">=1.0.2"
django-redis = ">=4.0.0"
django-storages = {extras = ["google"],version = "*"}
django-templated-email = ">=2.2.0"
django-versatileimagefield = ">=1.10"
django-webpack-loader = ">=0.3.0,<=0.6.0"
django-elasticsearch-dsl = ">=0.5.0.<0.6.0"
django-recaptcha = "<2.0.0"
django-silk = "==2.0.0"
django-debug-toolbar = "*"
django-debug-toolbar-request-history = "*"
django-extensions = "*"
ipython = "*"
elasticsearch = "==6.3.1"
elasticsearch-dsl = ">=6.0.0,<6.1.0"
faker = ">=0.7.7"
freezegun = ">=0.3.9"
google-measurement-protocol = ">=1.0.0"
"google-i18n-address" = ">=2.3.0"
graphene-django = ">=2.2,<=2.3.2"
Markdown = ">=2.4,<=3.1.1"
maxminddb = "*"
"maxminddb-geolite2" = "*"
phonenumberslite = ">=8.10.10"
prices = ">=1.0.0"
"psycopg2-binary" = ">=2.7,<=2.8.6"
purl = ">=0.4.1"
raven = "==6.9.0"
requests = ">=1.2.0"
social-auth-app-django = ">=2.0.0"
text-unidecode = "*"
uwsgi = {version = ">=2.0.0",platform_system = "!= 'Windows'"}
celery = {version = ">=4.3",extras = ["redis"]}
redis = "<4.0"
kombu = "<5.0.0"
weasyprint = ">=0.42.2"
graphene-django-optimizer = ">=0.4.0,<=0.4.2"
braintree = "==3.49.0"
razorpay = ">=1.1.1"
stripe = "*"
"urllib3" = "*"
dataclasses = {version = "*",markers = "python_version < '3.7'"}
django-contact-form = "*"
whitenoise = {extras = ["brotli"],version = "*"}

[dev-packages]
pytest = "*"
pytest-django = "*"
pytest-django-queries = "*"
pytest-vcr = "*"
pytest-mock = "*"
pytest-xdist = "*"
pylint-django = "*"
coverage = "*"
pytest-cov = "*"
codecov = "*"
tox = "*"
pycodestyle = "*"
pydocstyle = "*"
pylint = "*"
pylint_celery = "*"
isort = "*"
yapf = "*"
pylint-plugin-utils = "*"
transifex-client = "*"
black = "==19.3b0"
pre-commit = "*"
flake8 = "*"

I'm using pipenv, version 2022.4.8. Any suggestion how to solve this issue would be much appreciated.

[leirons]

I have more than 120 packages and it took about 1 hour, i thought the problem in me, but i was wrong, it's awfull use it in this state

[matteius]

@leirons Are you locking against pypi or a private package server? Private servers do not provide a hash API that is used to optimize the lock generation process.

[matteius]

@em1208 I just tried your example from within my Ubuntu VirtualBox VM (which is slower than my native machine) and it took 4.5 minutes. pipenv, version 2022.8.5

✔ Success! 
Updated Pipfile.lock (555f96)!

real	4m34.679s
user	3m21.142s
sys	0m8.162s

[matteius]

I am locking this discussion of this old and closed issue, and I am once again asking that if you are having a new issue that you don't respond to an old closed issue report with minimal details -- please open a new issue report and follow the template.

It is highly likely that your comment could be missed responding to an old issue as well, because we actively prioritize and tag new issues as they come in.