Pipenv install fails to lock on recursive sub dependencies from private repos #3328
Labels
Category: VCS
Relates to version control system dependencies.
Type: Bug 🐛
This issue is a bug.
Type: Vendored Dependencies
This issue affects vendored dependencies within pipenv.
Comments
|
I'm having the same issue. I'll just add that I'm using PEP508 syntax to include VCS depencencies in the Edit: One difference is that I experience a whole bunch of |
techalchemy
added
Type: Bug 🐛
|
This will be resolved when #3298 is merged, thanks for your patience Do note that |
fwojciak
pushed a commit
to fwojciak/pipenv
that referenced
this issue
May 29, 2020
2020.5.28 (2020-05-28) ====================== Features & Improvements ----------------------- - `pipenv install` and `pipenv sync` will no longer attempt to install satisfied dependencies during installation. pypa#3057, pypa#3506 - Added support for resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148 - Added full support for resolution of all dependency types including direct URLs, zip archives, tarballs, etc. - Improved error handling and formatting. - Introduced improved cross platform stream wrappers for better `stdout` and `stderr` consistency. pypa#3298 - For consistency with other commands and the `--dev` option description, `pipenv lock --requirements --dev` now emits both default and development dependencies. The new `--dev-only` option requests the previous behaviour (e.g. to generate a `dev-requirements.txt` file). pypa#3316 - Pipenv will now successfully recursively lock VCS sub-dependencies. pypa#3328 - Added support for `--verbose` output to `pipenv run`. pypa#3348 - Pipenv will now discover and resolve the intrinsic dependencies of **all** VCS dependencies, whether they are editable or not, to prevent resolution conflicts. pypa#3368 - Added a new environment variable, `PIPENV_RESOLVE_VCS`, to toggle dependency resolution off for non-editable VCS, file, and URL based dependencies. pypa#3577 - Added the ability for Windows users to enable emojis by setting `PIPENV_HIDE_EMOJIS=0`. pypa#3595 - Allow overriding `PIPENV_INSTALL_TIMEOUT` environment variable (in seconds). pypa#3652 - Allow overriding `PIP_EXISTS_ACTION` evironment variable (value is passed to pip install). Possible values here: <https://pip.pypa.io/en/stable/reference/pip/#exists-action-option> Useful when you need to `PIP\_EXISTS\_ACTION=i` (ignore existing packages) - great for CI environments, where you need really fast setup. pypa#3738 - Pipenv will no longer forcibly override `PIP_NO_DEPS` on all vcs and file dependencies as resolution happens on these in a pre-lock step. pypa#3763 - Improved verbose logging output during `pipenv lock` will now stream output to the console while maintaining a spinner. pypa#3810 - Added support for automatic python installs via `asdf` and associated `PIPENV_DONT_USE_ASDF` environment variable. pypa#4018 - Pyenv/asdf can now be used whether or not they are available on PATH. Setting `PYENV_ROOT`/`ASDF_DIR` in a `.env` file allows Pipenv to install an interpreter without any shell customizations, so long as pyenv/asdf is installed. pypa#4245 - Added `--key` command line parameter for including personal PyUp.io API tokens when running `pipenv check`. pypa#4257 Behavior Changes ---------------- - Make conservative checks of known exceptions when subprocess returns output, so user won\'t see the whole traceback - just the error. pypa#2553 - Do not touch Pipfile early and rely on it so that one can do `pipenv sync` without a Pipfile. pypa#3386 - Re-enable `--help` option for `pipenv run` command. pypa#3844 - Make sure `pipenv lock -r --pypi-mirror {MIRROR_URL}` will respect the pypi-mirror in requirements output. pypa#4199 Bug Fixes --------- - Raise `PipenvUsageError` when \[\[source\]\] does not contain url field. pypa#2373 - Fixed a bug which caused editable package resolution to sometimes fail with an unhelpful setuptools-related error message. pypa#2722 - Fixed an issue which caused errors due to reliance on the system utilities `which` and `where` which may not always exist on some systems. - Fixed a bug which caused periodic failures in python discovery when executables named `python` were not present on the target `$PATH`. pypa#2783 - Dependency resolution now writes hashes for local and remote files to the lockfile. pypa#3053 - Fixed a bug which prevented `pipenv graph` from correctly showing all dependencies when running from within `pipenv shell`. pypa#3071 - Fixed resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148 - Fixed a bug which caused failures in warning reporting when running pipenv inside a virtualenv under some circumstances. - Fixed a bug with package discovery when running `pipenv clean`. pypa#3298 - Quote command arguments with carets (`^`) on Windows to work around unintended shell escapes. pypa#3307 - Handle alternate names for UTF-8 encoding. pypa#3313 - Abort pipenv before adding the non-exist package to Pipfile. pypa#3318 - Don\'t normalize the package name user passes in. pypa#3324 - Fix a bug where custom virtualenv can not be activated with pipenv shell pypa#3339 - Fix a bug that `--site-packages` flag is not recognized. pypa#3351 - Fix a bug where `pipenv --clear` is not working pypa#3353 - Fix unhashable type error during `$ pipenv install --selective-upgrade` pypa#3384 - Dependencies with direct `PEP508` compliant VCS URLs specified in their `install_requires` will now be successfully locked during the resolution process. pypa#3396 - Fixed a keyerror which could occur when locking VCS dependencies in some cases. pypa#3404 - Fixed a bug that `ValidationError` is thrown when some fields are missing in source section. pypa#3427 - Updated the index names in lock file when source name in Pipfile is changed. pypa#3449 - Fixed an issue which caused `pipenv install --help` to show duplicate entries for `--pre`. pypa#3479 - Fix bug causing `[SSL: CERTIFICATE_VERIFY_FAILED]` when Pipfile `[[source]]` has `verify_ssl=false` and url with custom port. pypa#3502 - Fix `sync --sequential` ignoring `pip install` errors and logs. pypa#3537 - Fix the issue that lock file can\'t be created when `PIPENV_PIPFILE` is not under working directory. pypa#3584 - Pipenv will no longer inadvertently set `editable=True` on all vcs dependencies. pypa#3647 - The `--keep-outdated` argument to `pipenv install` and `pipenv lock` will now drop specifier constraints when encountering editable dependencies. - In addition, `--keep-outdated` will retain specifiers that would otherwise be dropped from any entries that have not been updated. pypa#3656 - Fixed a bug which sometimes caused pipenv to fail to respect the `--site-packages` flag when passed with `pipenv install`. pypa#3718 - Normalize the package names to lowercase when comparing used and in-Pipfile packages. pypa#3745 - `pipenv update --outdated` will now correctly handle comparisons between pre/post-releases and normal releases. pypa#3766 - Fixed a `KeyError` which could occur when pinning outdated VCS dependencies via `pipenv lock --keep-outdated`. pypa#3768 - Resolved an issue which caused resolution to fail when encountering poorly formatted `python_version` markers in `setup.py` and `setup.cfg` files. pypa#3786 - Fix a bug that installation errors are displayed as a list. pypa#3794 - Update `pythonfinder` to fix a problem that `python.exe` will be mistakenly chosen for virtualenv creation under WSL. pypa#3807 - Fixed several bugs which could prevent editable VCS dependencies from being installed into target environments, even when reporting successful installation. pypa#3809 - `pipenv check --system` should find the correct Python interpreter when `python` does not exist on the system. pypa#3819 - Resolve the symlinks when the path is absolute. pypa#3842 - Pass `--pre` and `--clear` options to `pipenv update --outdated`. pypa#3879 - Fixed a bug which prevented resolution of direct URL dependencies which have PEP508 style direct url VCS sub-dependencies with subdirectories. pypa#3976 - Honor `PIPENV_SPINNER` environment variable pypa#4045 - Fixed an issue with `pipenv check` failing due to an invalid API key from `pyup.io`. pypa#4188 - Fixed a bug which caused versions from VCS dependencies to be included in `Pipfile.lock` inadvertently. pypa#4217 - Fixed a bug which caused pipenv to search non-existent virtual environments for `pip` when installing using `--system`. pypa#4220 - `Requires-Python` values specifying constraint versions of python starting from `1.x` will now be parsed successfully. pypa#4226 - Fix a bug of `pipenv update --outdated` that can\'t print output correctly. pypa#4229 - Fixed a bug which caused pipenv to prefer source distributions over wheels from `PyPI` during the dependency resolution phase. Fixed an issue which prevented proper build isolation using `pep517` based builders during dependency resolution. pypa#4231 - Don\'t fallback to system Python when no matching Python version is found. pypa#4232 Vendored Libraries ------------------ - Updated `pip_shims` to support `--outdated` with new pip versions. pypa#3766 - Update vendored dependencies and invocations - Update vendored and patched dependencies - Update patches on `piptools`, `pip`, `pip-shims`, `tomlkit` - Fix invocations of dependencies - Fix custom `InstallCommand` instantiation - Update `PackageFinder` usage - Fix `Bool` stringify attempts from `tomlkit` - Updated vendored dependencies: - **attrs**: `18.2.0 => `19.1.0` - **certifi**: `2018.10.15 => `2019.3.9` - **cached\_property**: `1.4.3 => `1.5.1` - **cerberus**: `1.2.0 => `1.3.1` - **click**: `7.0.0 => `7.1.1` - **click-completion**: `0.5.0 => `0.5.1` - **colorama**: `0.3.9 => `0.4.3` - **contextlib2**: `(new) => `0.6.0.post1` - **distlib**: `0.2.8 => `0.2.9` - **funcsigs**: `(new) => `1.0.2` - **importlib\_metadata** `1.3.0 => `1.5.1` - **importlib-resources**: `(new) => `1.4.0` - **idna**: `2.7 => `2.9` - **jinja2**: `2.10.0 => `2.11.1` - **markupsafe**: `1.0 => `1.1.1` - **more-itertools**: `(new) => `5.0.0` - **orderedmultidict**: `(new) => `1.0` - **packaging**: `18.0 => `19.0` - **parse**: `1.9.0 => `1.15.0` - **pathlib2**: `2.3.2 => `2.3.3` - **pep517**: `(new) => `0.5.0` - **pexpect**: `4.6.0 => `4.8.0` - **pip-shims**: `0.2.0 => `0.5.1` - **pipdeptree**: `0.13.0 => `0.13.2` - **pyparsing**: `2.2.2 => `2.4.6` - **python-dotenv**: `0.9.1 => `0.10.2` - **pythonfinder**: `1.1.10 => `1.2.2` - **pytoml**: `(new) => `0.1.20` - **requests**: `2.20.1 => `2.23.0` - **requirementslib**: `1.3.3 => `1.5.4` - **scandir**: `1.9.0 => `1.10.0` - **shellingham**: `1.2.7 => `1.3.2` - **six**: `1.11.0 => `1.14.0` - **tomlkit**: `0.5.2 => `0.5.11` - **urllib3**: `1.24 => `1.25.8` - **vistir**: `0.3.0 => `0.5.0` - **yaspin**: `0.14.0 => `0.14.3` - **zipp**: `0.6.0` - Removed vendored dependency **cursor**. pypa#4169 - Add and update vendored dependencies to accommodate `safety` vendoring: - **safety** `(none)` => `1.8.7` - **dparse** `(none)` => `0.5.0` - **pyyaml** `(none)` => `5.3.1` - **urllib3** `1.25.8` => `1.25.9` - **certifi** `2019.11.28` => `2020.4.5.1` - **pyparsing** `2.4.6` => `2.4.7` - **resolvelib** `0.2.2` => `0.3.0` - **importlib-metadata** `1.5.1` => `1.6.0` - **pip-shims** `0.5.1` => `0.5.2` - **requirementslib** `1.5.5` => `1.5.6` pypa#4188 - Updated vendored `pip` => `20.0.2` and `pip-tools` => `5.0.0`. pypa#4215 - Updated vendored dependencies to latest versions for security and bug fixes: - **requirementslib** `1.5.8` => `1.5.9` - **vistir** `0.5.0` => `0.5.1` - **jinja2** `2.11.1` => `2.11.2` - **click** `7.1.1` => `7.1.2` - **dateutil** `(none)` => `2.8.1` - **backports.functools\_lru\_cache** `1.5.0` => `1.6.1` - **enum34** `1.1.6` => `1.1.10` - **toml** `0.10.0` => `0.10.1` - **importlib\_resources** `1.4.0` => `1.5.0` pypa#4226 - Changed attrs import path in vendored dependencies to always import from `pipenv.vendor`. pypa#4267 Improved Documentation ---------------------- - Added documenation about variable expansion in `Pipfile` entries. pypa#2317 - Consolidate all contributing docs in the rst file pypa#3120 - Update the out-dated manual page. pypa#3246 - Move CLI docs to its own page. pypa#3346 - Replace (non-existant) video on docs index.rst with equivalent gif. pypa#3499 - Clarify wording in Basic Usage example on using double quotes to escape shell redirection pypa#3522 - Ensure docs show navigation on small-screen devices pypa#3527 - Added a link to the TOML Spec under General Recommendations & Version Control to clarify how Pipfiles should be written. pypa#3629 - Updated the documentation with the new `pytest` entrypoint. pypa#3759 - Fix link to GIF in README.md demonstrating Pipenv\'s usage, and add descriptive alt text. pypa#3911 - Added a line describing potential issues in fancy extension. pypa#3912 - Documental description of how Pipfile works and association with Pipenv. pypa#3913 - Clarify the proper value of `python_version` and `python_full_version`. pypa#3914 - Write description for `--deploy` extension and few extensions differences. pypa#3915 - More documentation for `.env` files pypa#4100 - Updated documentation to point to working links. pypa#4137 - Replace docs.pipenv.org with pipenv.pypa.io pypa#4167 - Added functionality to check spelling in documentation and cleaned up existing typographical issues. pypa#4209
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This issue is very similar to the closed issue #920, I just added recursive to the title and changed install to lock.
Issue description
pipenv fails during locking to resolve a private dependency of a private dependency. By private I mean the URL/git-repo is specified via
dependency_linksinsetup.pyand not (yet) available on pipy.The module structure is:
demo_lib: no dependencydemo_measure: depends ondemo_libdemo_paper: depends ondemo_measurerunning
PIP_PROCESS_DEPENDENCY_LINKS=1 pipenv install -e .in projectdemo_libordemo_measure(the situation of issue #920) works fine, but running it indemo_paperyields a resolve error fordemo_lib.Expected result
It should work like
pip install -e . --process-dependency-links.Not sure if it easy to fix, since one needs to inspect the
dependency_linksof allsetup.pyof all dependencies, not just the current project's one. But from installing, it should be possible to track the source (i.e. the private git repo specified independency_links).Running my own pip-server is not impossible, but it's just so convenient to use
dependency_linksduring closed dev and then just drop it insetup.pywhen published.Actual result
install is fine, output:
locking fails, output:
Steps to replicate
I made the three dummy-repos on my gitlab public for reproducibility (https://gitlab.fubar.science/mskoenz/demo_paper)
git clone git@gitlab.fubar.science:mskoenz/demo_paper.gitcd demo_paperPIP_PROCESS_DEPENDENCY_LINKS=1 pipenv install -e . -v$ pipenv --support
Pipenv version:
'2018.10.13'Pipenv location:
'/Users/mskoenz/.pyenv/versions/3.7.0/lib/python3.7/site-packages/pipenv'Python location:
'/Users/mskoenz/.pyenv/versions/3.7.0/bin/python3'Python installations found:
3.7.0:/Users/mskoenz/.pyenv/shims/python33.7.0:/Users/mskoenz/.pyenv/shims/python3.7.0:/Users/mskoenz/.pyenv/shims/python3.7m3.7.0:/Users/mskoenz/.pyenv/shims/python3.73.7.0:/Users/mskoenz/.pyenv/versions/3.7.0/bin/python3.73.7.0:/Users/mskoenz/.pyenv/versions/3.7.0/bin/python3.7m3.7.0:/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/bin/python3.73.7.0:/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/bin/python3.7m3.6.6:/Users/mskoenz/.pyenv/shims/python3.63.6.6:/Users/mskoenz/.pyenv/shims/python3.6m3.6.6:/Users/mskoenz/.pyenv/versions/3.6.6/bin/python3.63.6.6:/Users/mskoenz/.pyenv/versions/3.6.6/bin/python3.6m3.5.6:/Users/mskoenz/.pyenv/shims/python3.5m3.5.6:/Users/mskoenz/.pyenv/shims/python3.53.5.6:/Users/mskoenz/.pyenv/versions/3.5.6/bin/python3.53.5.6:/Users/mskoenz/.pyenv/versions/3.5.6/bin/python3.5m3.4.9:/Users/mskoenz/.pyenv/shims/python3.4m3.4.9:/Users/mskoenz/.pyenv/shims/python3.43.4.9:/Users/mskoenz/.pyenv/versions/3.4.9/bin/python3.43.4.9:/Users/mskoenz/.pyenv/versions/3.4.9/bin/python3.4m2.7.15:/Users/mskoenz/.pyenv/shims/python22.7.15:/Users/mskoenz/.pyenv/shims/python2.72.7.15:/Users/mskoenz/.pyenv/versions/2.7.15/bin/python2.72.7.15:/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/bin/python2.72.7.10:/Users/mskoenz/.pyenv/versions/2.7.10/bin/python2.72.7.10:/usr/bin/python2.7.10:/usr/bin/pythonw2.7.10:/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7PEP 508 Information:
System environment variables:
CPLUS_INCLUDE_PATHLDFLAGSVIRTUALENVWRAPPER_SCRIPTVIRTUALENVWRAPPER_PROJECT_FILENAMETERM_PROGRAMPYENV_ROOTSHELLTERMCPPFLAGSTMPDIRPERL5LIBApple_PubSub_Socket_RenderPERL_MB_OPTPYENV_VERSIONUSERSSH_AUTH_SOCKPYENV_DIR__CF_USER_TEXT_ENCODINGWORKON_HOMEVIRTUALENVWRAPPER_PYTHONPATHVIRTUALENVWRAPPER_HOOK_DIRC_INCLUDE_PATHPWDLANGPYENV_HOOK_PATHXPC_FLAGSXPC_SERVICE_NAMEPYENV_SHELLSHLVLHOMEPERL_LOCAL_LIB_ROOTLOGNAMEVIRTUALENVWRAPPER_WORKON_CDPERL_MM_OPTPYENV_VIRTUALENVWRAPPER_PREFER_PYVENVPYTHONDONTWRITEBYTECODEPIP_SHIMS_BASE_MODULEPIP_PYTHON_PATHPipenv–specific environment variables:
Debug–specific environment variables:
PATH:/Users/mskoenz/.pyenv/versions/3.7.0/bin:/usr/local/Cellar/pyenv/1.2.7/libexec:/Users/mskoenz/.pyenv/plugins/pyenv-install-latest/bin:/Users/mskoenz/perl5/bin:/Users/mskoenz/.pyenv/shims:/usr/local/opt/sqlite/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Users/mskoenz/perl5/bin:/Users/mskoenz/.pyenv/shims:/usr/local/opt/sqlite/bin:/usr/local/opt/coreutils/libexec/gnubinSHELL:/bin/bashLANG:en_US.UTF-8PWD:/Users/mskoenz/Desktop/workbench/install_test/demo_paperContents of
Pipfile('/Users/mskoenz/Desktop/workbench/install_test/demo_paper/Pipfile'):The text was updated successfully, but these errors were encountered: