Pipenv does not recursively lock dependencies of packages excluded by markers

[axnsan12]

With a Pipfile that contains packages restricted by environment markers, running pipenv lock from an excluded environment will not recursively lock the excluded packages' dependencies.

$ python -m pipenv.help output

Pipenv version: '2018.6.25'

Pipenv location: 'C:\\Python36\\lib\\site-packages\\pipenv'

Python location: 'C:\\Python36\\python.exe'

Other Python installations in PATH:

• 2.7: C:\Python27\python2.7.exe

• 2.7: C:\Python27\python2.7.exe

• 2.7: C:\cygwin64\bin\python2.7.exe

• 3.6: C:\Python36\python.exe

• 3.6.5: C:\Python36\python.exe

• 2.7.12: C:\Python27\python.exe

• None: C:\cygwin64\bin\python

• 2.7.12: C:\Python27\python2.exe

• None: C:\cygwin64\bin\python2

• 3.6.5: C:\Windows\py.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.17134',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'win32'}

System environment variables:

• ALLUSERSPROFILE
• ANDROID_SDK_HOME
• ANSICON
• ANSICON_DEF
• ANT_HOME
• APPDATA
• CHOCOLATEYINSTALL
• CHOCOLATEYLASTPATHUPDATE
• CLINK_DIR
• COMMONPROGRAMFILES
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• COMPUTERNAME
• COMSPEC
• CONEMUANSI
• CONEMUANSILOG
• CONEMUARGS
• CONEMUARGS2
• CONEMUBACKHWND
• CONEMUBASEDIR
• CONEMUBUILD
• CONEMUCONFIG
• CONEMUDIR
• CONEMUDRAWHWND
• CONEMUDRIVE
• CONEMUHOOKS
• CONEMUHWND
• CONEMUPALETTE
• CONEMUPID
• CONEMUSERVERPID
• CONEMUTASK
• CONEMUWORKDIR
• CONEMUWORKDRIVE
• CONNECTIONSTRING
• DOKANLIBRARY1
• DRIVERDATA
• FARHOME
• FARLANG
• FARLOCALPROFILE
• FARPROFILE
• FPS_BROWSER_APP_PROFILE_STRING
• FPS_BROWSER_USER_PROFILE_STRING
• FSHARPINSTALLDIR
• GOOGLE_APPLICATION_CREDENTIALS
• GOPATH
• GOROOT
• HOMEDRIVE
• HOMEPATH
• INTEL_DEV_REDIST
• LOCALAPPDATA
• LOGONSERVER
• MIC_LD_LIBRARY_PATH
• MYSQLCONNECTOR_ASSEMBLIESPATH
• NDI_RUNTIME_DIR_V2
• NDI_RUNTIME_DIR_V3
• NUMBER_OF_PROCESSORS
• ONEDRIVE
• OS
• PATH
• PATHEXT
• PIPENV_DEFAULT_PYTHON_VERSION
• PIPENV_SHELL_FANCY
• PIPENV_VENV_IN_PROJECT
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMDATA
• PROGRAMFILES
• PROGRAMFILES(X86)
• PROGRAMW6432
• PROMPT
• PSMODULEPATH
• PT7HOME
• PUBLIC
• RUBYOPT
• SESSIONNAME
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TMP
• TVT
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• VBOX_MSI_INSTALL_PATH
• VS140COMNTOOLS
• WINDIR
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

• PIPENV_DEFAULT_PYTHON_VERSION: 3.6
• PIPENV_SHELL_FANCY: true
• PIPENV_VENV_IN_PROJECT: true

Debug–specific environment variables:

• PATH: C:\Program Files\Far Manager\ConEmu\Scripts;C:\Program Files\Far Manager;C:\Program Files\Far Manager\ConEmu;C:\Program Files (x86)\Graphviz2.38\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files\Docker\Docker\Resources\bin;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler;C:\ProgramData\Oracle\Java\javapath;C:\Python36\Scripts\;C:\Python36\;C:\Python27\Scripts;C:\Python27;C:\bin\imagemagick;C:\Program Files\MongoDB\Server\3.4\bin;C:\cygwin64\bin;C:\Program Files (x86)\PuTTY\;C:\bin;C:\Android\sdk\platform-tools;C:\Android\sdk\tools;C:\Users\axnsan\Application Data\npm;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\ProgramData\chocolatey\bin;C:\Program Files\OpenVPN\bin;C:\Program Files\dotnet\;C:\Program Files (x86)\IDA 6.8;C:\Program Files\Calibre2\;C:\Windows\System32;C:\Go\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\nodejs\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\texlive\2018\bin\win32;C:\Ruby24-x64\bin;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Program Files\OpenVPN\bin;C:\Program Files (x86)\PuTTY\;C:\Program Files\Git\cmd;C:\Python27\Scripts;C:\Python27;C:\bin;C:\Android\sdk\platform-tools;C:\Android\sdk\tools;C:\Program Files\nodejs\;C:\Users\axnsan\Application Data\npm;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Users\axnsan\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Diffuse;C:\Users\axnsan\AppData\Roaming\npm;C:\Users\axnsan\AppData\Local\GitHubDesktop\bin;C:\Users\axnsan\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\EaseUS\Todo Backup\bin\x64\

---

Contents of Pipfile ('C:\Projects\pipenv-excluded-lock\Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.6"

[packages]
django-auth-ldap = {version = ">=1.6.1", sys_platform = "!= 'win32'"}

Contents of Pipfile.lock ('C:\Projects\pipenv-excluded-lock\Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "bce05c2960fbf5ea379c5782007507b8cb98be9b9054f1a59b425ab719c2149e"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "django-auth-ldap": {
            "hashes": [
                "sha256:3661287d9a57755c65a5377c80ec1efbcea9146f01050b9370f248177c113415",
                "sha256:6cbf45ec36bb67893923e94d0d0a65bb8773451a136570876bf5362adcc67f4b"
            ],
            "index": "pypi",
            "markers": "sys_platform != 'win32'",
            "version": "==1.6.1"
        }
    },
    "develop": {}
}

---

Expected result

Dependencies of packages are locked so that they properly install on the targeted environment. It works properly when you remove the markers or lock under a matching environment.

Locking [dev-packages] dependencies...
Locking [packages] dependencies...
using sources: [{'url': 'https://pypi.python.org/simple', 'verify_ssl': True, 'name': 'pypi'}]
Using pip: -i https://pypi.python.org/simple

                          ROUND 1                           
Current constraints:
  django-auth-ldap>=1.6.1 (from -r C:\Users\axnsan\AppData\Local\Temp\pipenv-ge799h7q-requirements\pipenv-o9jwj3f7-constraints.txt (line 2))

Finding the best candidates:
  found candidate django-auth-ldap==1.6.1 (constraint was >=1.6.1)

Finding secondary dependencies:
  django-auth-ldap==1.6.1 not in cache, need to check index
  django-auth-ldap==1.6.1   requires django-auth-ldap==1.6.1, django>=1.11, pyasn1>=0.3.7, pyasn1_modules>=0.1.5, python-ldap>=3.1, pytz

New dependencies found in this round:
  adding ['django', '>=1.11', '[]']
  adding ['django-auth-ldap', '==1.6.1', '[]']
  adding ['pyasn1', '>=0.3.7', '[]']
  adding ['pyasn1-modules', '>=0.1.5', '[]']
  adding ['python-ldap', '>=3.1', '[]']
  adding ['pytz', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  django>=1.11
  django-auth-ldap==1.6.1,>=1.6.1 (from -r C:\Users\axnsan\AppData\Local\Temp\pipenv-ge799h7q-requirements\pipenv-o9jwj3f7-constraints.txt (line 2))
  pyasn1>=0.3.7
  pyasn1_modules>=0.1.5
  python-ldap>=3.1
  pytz

Finding the best candidates:
  found candidate django==2.0.6 (constraint was >=1.11)
  found candidate django-auth-ldap==1.6.1 (constraint was >=1.6.1,==1.6.1)
  found candidate pyasn1==0.4.3 (constraint was >=0.3.7)
  found candidate pyasn1-modules==0.2.1 (constraint was >=0.1.5)
  found candidate python-ldap==3.1.0 (constraint was >=3.1)
  found candidate pytz==2018.4 (constraint was <any>)

Finding secondary dependencies:
  python-ldap==3.1.0 not in cache, need to check index
  python-ldap==3.1.0        requires pyasn1>=0.3.7, pyasn1_modules>=0.1.5, python-ldap==3.1.0
  django==2.0.6             requires django==2.0.6, pytz
  pytz==2018.4              requires pytz==2018.4
  pyasn1==0.4.3 not in cache, need to check index
  pyasn1==0.4.3             requires pyasn1==0.4.3
  pyasn1-modules==0.2.1 not in cache, need to check index
  pyasn1-modules==0.2.1     requires pyasn1-modules==0.2.1, pyasn1<0.5.0,>=0.4.1
  django-auth-ldap==1.6.1   requires django-auth-ldap==1.6.1, django>=1.11, pyasn1>=0.3.7, pyasn1_modules>=0.1.5, python-ldap>=3.1, pytz

New dependencies found in this round:
  adding ['django', '==2.0.6,>=1.11', '[]']
  adding ['pyasn1', '<0.5.0,==0.4.3,>=0.3.7,>=0.4.1', '[]']
  adding ['pyasn1-modules', '==0.2.1,>=0.1.5', '[]']
  adding ['python-ldap', '==3.1.0,>=3.1', '[]']
  adding ['pytz', '==2018.4', '[]']
Removed dependencies in this round:
  removing ['django', '>=1.11', '[]']
  removing ['pyasn1', '>=0.3.7', '[]']
  removing ['pyasn1-modules', '>=0.1.5', '[]']
  removing ['python-ldap', '>=3.1', '[]']
  removing ['pytz', '', '[]']
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 2: not stable

                          ROUND 3                           
Current constraints:
  django==2.0.6,>=1.11
  django-auth-ldap==1.6.1,>=1.6.1 (from -r C:\Users\axnsan\AppData\Local\Temp\pipenv-ge799h7q-requirements\pipenv-o9jwj3f7-constraints.txt (line 2))
  pyasn1<0.5.0,==0.4.3,>=0.3.7,>=0.4.1
  pyasn1_modules==0.2.1,>=0.1.5
  python-ldap==3.1.0,>=3.1
  pytz==2018.4

Finding the best candidates:
  found candidate django==2.0.6 (constraint was >=1.11,==2.0.6)
  found candidate django-auth-ldap==1.6.1 (constraint was >=1.6.1,==1.6.1)
  found candidate pyasn1==0.4.3 (constraint was >=0.3.7,>=0.4.1,==0.4.3,<0.5.0)
  found candidate pyasn1-modules==0.2.1 (constraint was >=0.1.5,==0.2.1)
  found candidate python-ldap==3.1.0 (constraint was >=3.1,==3.1.0)
  found candidate pytz==2018.4 (constraint was ==2018.4)

Finding secondary dependencies:
  django==2.0.6             requires django==2.0.6, pytz
  python-ldap==3.1.0        requires pyasn1>=0.3.7, pyasn1_modules>=0.1.5, python-ldap==3.1.0
  pyasn1==0.4.3             requires pyasn1==0.4.3
  pyasn1-modules==0.2.1     requires pyasn1-modules==0.2.1, pyasn1<0.5.0,>=0.4.1
  django-auth-ldap==1.6.1   requires django-auth-ldap==1.6.1, django>=1.11, pyasn1>=0.3.7, pyasn1_modules>=0.1.5, python-ldap>=3.1, pytz
  pytz==2018.4              requires pytz==2018.4
------------------------------------------------------------
Result of round 3: stable, done

Updated Pipfile.lock (37f0bc)!

Actual result

Only the top-level package is locked.

Creating a virtualenv for this project...
Pipfile: C:\Projects\pipenv-excluded-lock\Pipfile
Using C:\Python36\python.exe (3.6.5) to create virtualenv...
Running virtualenv with interpreter C:\Python36\python.exe

Using base prefix 'C:\\Python36'

New python executable in C:\Projects\pipenv-excluded-lock\.venv\Scripts\python.exe

Installing setuptools, pip, wheel...done.


Virtualenv location: C:\Projects\pipenv-excluded-lock\.venv
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
using sources: [{'url': 'https://pypi.python.org/simple', 'verify_ssl': True, 'name': 'pypi'}]
Using pip: -i https://pypi.python.org/simple

                          ROUND 1                           
Current constraints:
  django-auth-ldap>=1.6.1 (from -r C:\Users\axnsan\AppData\Local\Temp\pipenv-yd3bfpqg-requirements\pipenv-mga4xrpk-constraints.txt (line 2))

Finding the best candidates:
  found candidate django-auth-ldap==1.6.1 (constraint was >=1.6.1)

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Updated Pipfile.lock (c2149e)!

{
    "_meta": {
        "hash": {
            "sha256": "bce05c2960fbf5ea379c5782007507b8cb98be9b9054f1a59b425ab719c2149e"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "django-auth-ldap": {
            "hashes": [
                "sha256:3661287d9a57755c65a5377c80ec1efbcea9146f01050b9370f248177c113415",
                "sha256:6cbf45ec36bb67893923e94d0d0a65bb8773451a136570876bf5362adcc67f4b"
            ],
            "index": "pypi",
            "markers": "sys_platform != 'win32'",
            "version": "==1.6.1"
        }
    },
    "develop": {}
}

Steps to replicate

• create Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.6"

[packages]
django-auth-ldap = {version = ">=1.6.1", sys_platform = "!= 'win32'"}

• under Windows, run pipenv lock
• get output as outlined above
• run pipenv install --ignore-pipfile on linux using the new Pipenv.lock file
• cry as you watch that django-auth-ldap was installed without its dependencies

[techalchemy]

Right, we specifically can't ignore top level markers during resolution -- we used to ignore them, but it causes serious issues for things that legitimately can't be installed in the specified environment. For instance, if I say pywin32 = {markers = "os_name == 'nt'"}, but pipenv decides to ignore my markers and attempt to resolve anyway, the resolution process explodes in my face on any platform besides windows.

That example is not contrived, that's a real example, we had a lot of bug reports on just that one specific one. Another example is if you put python_version markers on a package-- say you specify something can only be installed for python 3 or higher. Try locking against a python 2 environment. If we ignore your markers, this explodes.

Essentially, if you want something to be resolved into your lockfile, it needs to evaluate as installable in the given environment. This is the nature of python packaging -- if we are told we can't install something we won't attempt to, because if we attempt to run a setup.py that is not meant for the environment we are running in, we have no guarantees of success.

[gsemet]

So, what is the solution to have pipenv happy with {markers = "os_name == 'nt'"} and no attempting to install this dependencies under linux ? I still have this annoying "Could not find a version that matches pywin32>=223" where I explicitely told pipenv to ignore it. Or the markers features is useless and should be removed.

[techalchemy]

@gsemet thanks for that incredibly helpful feedback. See #2584 #2359 and #2384.

Also, even if markers worked as you describe, just because they don’t accomplish a particular thing for you doesn’t mean they are useless and should be removed. Check your attitude before posting on the tracker. You are making a lot of assumptions.

[gsemet]

Hello. I am sorry to have appeared rude. That wasn't my intention. I see my markers being ignored on recent versions while they weren't before and interpret that as a regression. But looking as these closed tickets I though the behavior of pipenv was as expected, while I hope it is not. Markers are really important in Pipfile. They may be weird to see in the lock file, since the lock is the ultimate source of truth in a particular environment.

The problem is maybe it is hard to understand for us, users, the delay beween when a ticket is closed and when it is released. Maybe a label "released" ? Or the "milestone" feature of Github?

My use case:

• typing should not be installed on python > 3.5
• pynsist should only be installed on windows
• pywin32 as well. I understand this particular one may have an issue in it, but it worked on pipenv 2018.5.

[techalchemy]

Markers are really important in Pipfile. They may be weird to see in the lock file, since the lock is the ultimate source of truth in a particular environment.

we know. Thats why I linked you to the bug reports and the fixes. I don't understand how this is possibly interpreted as me telling you this behavior is expected. I don't really need more explanation of what you're doing, this is not complicated. There were some bugs. They have issues, fixes, and dates on them, and yes, even milestones often. Sometimes, we work on code and make changes before releasing it. This is after all github, a place for development of code. We use it to collaborate on this project.

If you are not sure if your issue is fixed, install from master and check.

[vi9sion]

Can I add piping install mysqlclient after a piping lock?