pipenv lock --keep-outdated updates subdependencies #3975

MartinAltmayer · 2019-10-04T14:45:08Z

Issue description

pipenv lock --keep-outdated updates versions of subdependencies despite the --keep-outdated flag.

Example: Pipfile contains python-dateutils which has six as a dependency. If a newer version of six is available it will be put into Pipfile.lock by pipenv lock --keep-outdated.

Expected result

When used with --keep-outdated, pipenv lock uses the currently installed versions without doing any updates.

Actual result

Pipenv lock puts newer versions of subdependencies into Pipfile.lock.

Steps to replicate

Put this Pipfile into an empty folder

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
python-dateutil = "==2.7"
six = "==1.5"

[requires]
python_version = "3.7"

pipenv install
Pipfile.lock will contain `six = '==1.5'
delete the line `six = '==1.5' from the Pipfile (this simulates a release of a newer version of a subdependency)
pipenv lock --keep-outdated
Pipfile.lock will now contain six = '==1.12'.

This issue seems to apply only to subdependencsies: If you specify "*" for python-dateutil in the Pipfile, pipenv lock --keep-outdated will (correctly) refrain from updating python-dateutil.

$ pipenv --support

Pipenv version: '2018.11.15.dev0'

Pipenv location: '/usr/lib/python3.7/site-packages/pipenv'

Python location: '/usr/bin/python'

Python installations found:

3.7.4: /usr/bin/python3
3.7.4: /usr/bin/python3.7m

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_system': 'Linux',
 'python_full_version': '3.7.4',
 'python_version': '3.7',
 'sys_platform': 'linux'}

System environment variables:

Pipenv–specific environment variables:

Debug–specific environment variables:

Contents of Pipfile ('/home/martin/temp/piptest/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
python-dateutil = "==2.7"

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('/home/martin/temp/piptest/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "458ffc4c79d86d5ffe6888bcc057e0dfdf5ef89c0728cd0ea600dfebb33042dd"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "python-dateutil": {
            "hashes": [
                "sha256:07009062406cffd554a9b4135cd2ff167c9bf6b7aac61fe946c93e69fad1bbd8",
                "sha256:8f95bb7e6edbb2456a51a1fb58c8dca942024b4f5844cae62c90aa88afe6e300"
            ],
            "index": "pypi",
            "version": "==2.7"
        },
        "six": {
            "hashes": [
                "sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
                "sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
            ],
            "version": "==1.12.0"
        }
    },
    "develop": {}
}

The text was updated successfully, but these errors were encountered:

DusanMadar · 2019-11-20T10:38:32Z

I can confirm the same behavior.

Tested a couple of scenarios

cp Pipfile.lock Pipfile.lock.original && pipenv lock --keep-outdated
- Pipfile.lock different from Pipfile.lock.original
mv Pipfile.lock Pipfile.lock.keep_deps && pipenv lock
- Pipfile.lock different from Pipfile.lock.original
- Pipfile.lock different from Pipfile.lock.keep_deps

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/usr/local/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

3.7.5: /usr/local/bin/python
3.7.5: /usr/local/bin/python3.7m

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.0.0-36-generic',
 'platform_system': 'Linux',
 'platform_version': '#39~18.04.1-Ubuntu SMP Tue Nov 12 11:09:50 UTC 2019',
 'python_full_version': '3.7.5',
 'python_version': '3.7',
 'sys_platform': 'linux'}

The OS is actually Alpine Linux 3.10.3 (Dockerfile FROM python:3.7-alpine)

mcallaghan-bsm · 2019-12-19T15:39:45Z

Can confirm this as well. It is super undesired. I think the key issue occurs when application/default packages share sub-dependencies with dev/develop packages.

The only WORKAROUND I found was to manually revert the "default" JSON section (short of figuring out some complex bash/git-patch script).

jacobeatsspam · 2020-07-10T16:12:15Z

@frostming Could this get some attention?

bendem · 2020-11-18T14:48:42Z

The problem makes sense, the goal of the lock is to update dependencies, the goal of pipenv lock --keep-outdated -d -r > requirements.txt is the opposite of that, we basically bastardized that command to do the opposite of what it was written for. Shouldn't that feature be moved to the sync command instead? sync reads the lockfile and installs dependencies, it could get a -r that writes a requirements file instead of installing.

tilgovi · 2020-11-18T19:22:07Z

The goal of lock is to generate a Pipfile.lock so that the current environment (working, hopefully) can be reproduced later. Ideally, this would generate a lock file that reflects the current state of installed packages by looking at what is actually installed right now. Nothing should be installed or upgraded unless it's currently missing entirely and therefore needs to be resolved.

The workflow that is still challenging is going from a Pipfile with semver ranges to a Pipfile.lock that pins the particular versions installed now, and it should be possible to selectively upgrade some dependencies and transitive dependencies and to cement those changes in the lock file without having other things updated unexpectedly.

frostming · 2020-11-19T01:52:10Z

Repeating the exact same steps on 2020.11.15, the issue has gone.

I am going to close this one, other comments are different issues, please track in #4539 . Thanks

bendem · 2020-11-19T10:32:01Z

The goal of lock is to generate a Pipfile.lock

Exactly my point, this feature requested is something that, if it updates the lockfile, is meaningless. The goal of the lock command is to resolve versions, this feature request is to never ever try to resolve versions but instead use the lockfile with all the versions already resolved.

tilgovi · 2020-11-19T18:11:48Z

The goal of the lock command is to resolve versions

I think this is the point of disagreement. The only time I expect it to resolve a version is when there is no existing lock file or the existing lock file entry does not satisfy an updated entry in the Pipfile. The request for this issue was to make it possible to make the minimal updates to satisfy the Pipfile, while keeping everything that already does satisfy unchanged.

mcallaghan-bsm · 2020-11-20T14:49:26Z

uhhhh

preconditions

sudo apt install python3.7-minimal
pip3 install pipenv

priming initial conditions

~/projects/pipenv_3975$ python3 -m pipenv install

$ cat Pipfile.lock 
{
    "_meta": {
        "hash": {
            "sha256": "3c64c9c09d36cfc9046ec8703b8bb5b757f7196e39c597ead4fb4e749b8b0332"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "python-dateutil": {
            "hashes": [
                "sha256:07009062406cffd554a9b4135cd2ff167c9bf6b7aac61fe946c93e69fad1bbd8",
                "sha256:8f95bb7e6edbb2456a51a1fb58c8dca942024b4f5844cae62c90aa88afe6e300"
            ],
            "index": "pypi",
            "version": "==2.7"
        },
        "six": {
            "hashes": [
                "sha256:7ba77770fedd5b84d6c67283912cad39856b5532dc1beb76182ccacee9ec41e0",
                "sha256:9fa05de9aa4e0042767dd666ce4905a097306b6291148e958162c4ca0b52d16d"
            ],
            "index": "pypi",
            "version": "==1.5"
        }
    },
    "develop": {}
}

then conduct the test, REMOVE six==1.5 from Pipfile
, and then try:

$ python3 -m pipenv lock --keep-outdated
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

Traceback (most recent call last):
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 760, in main
    dev=parsed.dev)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 712, in resolve_packages
    results = clean_outdated(results, resolver, project, dev)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 606, in clean_outdated
    reverse_deps = project.environment.reverse_dependencies()
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/project.py", line 376, in environment
    self._environment = self.get_environment(allow_global=allow_global)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/project.py", line 366, in get_environment
    environment.extend_dists(pipenv_dist)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/environment.py", line 127, in extend_dists
    extras = self.resolve_dist(dist, self.base_working_set)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/environment.py", line 122, in resolve_dist
    deps |= cls.resolve_dist(dist, working_set)
  File "/home/USER_SNIPPED/.local/lib/python3.6/site-packages/pipenv/environment.py", line 121, in resolve_dist
    dist = working_set.find(req)
  File "/home/USER_SNIPPED/.local/share/virtualenvs/pipenv_3975-PaIxvLxT/lib/python3.7/site-packages/pkg_resources/__init__.py", line 627, in find
    raise VersionConflict(dist, req)
pkg_resources.VersionConflict: (six 1.5.0 (/home/USER_SNIPPED/.local/share/virtualenvs/pipenv_3975-PaIxvLxT/lib/python3.7/site-packages), Requirement.parse('six<2,>=1.9.0'))

jacobeatsspam · 2020-11-23T16:00:30Z

@mcallaghan-bsm Your initial command is to install 3.7, but your trace is using 3.6. I think PEBKAC in this case.

mcallaghan-bsm · 2020-11-23T16:10:05Z

(kk abandoning issue closed NWA)

matteius · 2022-04-22T03:57:45Z

FWIW new versions of pipenv have a requirements command to generate the requirements.txt and this will likely eventually completely replace the pipenv lock -r behavior or at least that could lock and then run the requirements command, but the requirements command simply introspects the Pipfile.lock. Including hashes is one of the flags available for the new command.

triage-new-issues bot added the triage label Oct 4, 2019

mcallaghan-bsm mentioned this issue Dec 19, 2019

pipenv lock --keep-outdated does not respect Pipfile.lock constraints #4031

Closed

paytonrules mentioned this issue Apr 8, 2020

Generate requirements.txt from Pipfile.lock #3493

Closed

thundergolfer mentioned this issue Oct 13, 2020

Ability to lock down dependencies for reproducibility? bazelbuild/rules_python#140

Closed

jacobeatsspam mentioned this issue Nov 18, 2020

Creating requirements.txt re-calculates the lock file #4539

Closed

frostming closed this as completed Nov 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pipenv lock --keep-outdated updates subdependencies #3975

pipenv lock --keep-outdated updates subdependencies #3975

MartinAltmayer commented Oct 4, 2019

DusanMadar commented Nov 20, 2019 •

edited

Loading

mcallaghan-bsm commented Dec 19, 2019

jacobeatsspam commented Jul 10, 2020

bendem commented Nov 18, 2020

tilgovi commented Nov 18, 2020

frostming commented Nov 19, 2020 •

edited

Loading

bendem commented Nov 19, 2020

tilgovi commented Nov 19, 2020

mcallaghan-bsm commented Nov 20, 2020 •

edited

Loading

jacobeatsspam commented Nov 23, 2020

mcallaghan-bsm commented Nov 23, 2020

matteius commented Apr 22, 2022

pipenv lock --keep-outdated updates subdependencies #3975

pipenv lock --keep-outdated updates subdependencies #3975

Comments

MartinAltmayer commented Oct 4, 2019

Issue description

Expected result

Actual result

Steps to replicate

DusanMadar commented Nov 20, 2019 • edited Loading

mcallaghan-bsm commented Dec 19, 2019

jacobeatsspam commented Jul 10, 2020

bendem commented Nov 18, 2020

tilgovi commented Nov 18, 2020

frostming commented Nov 19, 2020 • edited Loading

bendem commented Nov 19, 2020

tilgovi commented Nov 19, 2020

mcallaghan-bsm commented Nov 20, 2020 • edited Loading

jacobeatsspam commented Nov 23, 2020

mcallaghan-bsm commented Nov 23, 2020

matteius commented Apr 22, 2022

DusanMadar commented Nov 20, 2019 •

edited

Loading

frostming commented Nov 19, 2020 •

edited

Loading

mcallaghan-bsm commented Nov 20, 2020 •

edited

Loading