Not all site-packages available when installed with --site-packages

[vlad0337187]

Issue description

If to create env with '--site-packages' option, no system packages are available in environment.

Expected result

System packages available in virtual environment.

Actual result

No system packages are available in virtualenv.
In system I have installed xdg package (from apt python3-xdg), it's placed in /usr/lib/python3/dist-packages/xdg.
I can import it from system python.
Can't import it from python inside pipenv environment.

python3.6 -m site in system:

sys.path = [
    '/home/vlad/Programs/organizing/gtg',
    '/usr/lib/python36.zip',
    '/usr/lib/python3.6',
    '/usr/lib/python3.6/lib-dynload',
    '/home/vlad/.local/lib/python3.6/site-packages',
    '/usr/local/lib/python3.6/dist-packages',
    '/usr/lib/python3/dist-packages',
    '/usr/lib/python3.6/dist-packages',
]
USER_BASE: '/home/vlad/.local' (exists)
USER_SITE: '/home/vlad/.local/lib/python3.6/site-packages' (exists)
ENABLE_USER_SITE: True

python -m site in pipenv virtualenv:

sys.path = [
    '/home/vlad/Programs/organizing/gtg',
    '/home/vlad/.local/share/virtualenvs/gtg-cnQmc6C0/lib/python36.zip',
    '/home/vlad/.local/share/virtualenvs/gtg-cnQmc6C0/lib/python3.6',
    '/home/vlad/.local/share/virtualenvs/gtg-cnQmc6C0/lib/python3.6/lib-dynload',
    '/usr/lib/python3.6',
    '/home/vlad/.local/share/virtualenvs/gtg-cnQmc6C0/lib/python3.6/site-packages',
]
USER_BASE: '/home/vlad/.local' (exists)
USER_SITE: '/home/vlad/.local/lib/python3.6/site-packages' (exists)
ENABLE_USER_SITE: False

System: Mint 19.1 (Ubuntu 16.04).
Used Python: 3.6 (system default)

Steps to replicate

pipenv --site-packages install
pipenv shell
python
import xdg # will throw import error. Same in sytem won't

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

Can't run the following (xclip not installed), so runned it without xclip:

$ pipenv --support | xclip

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/home/vlad/.local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

• 3.7.3: /usr/bin/python3.7
• 3.7.3: /usr/bin/python3.7m
• 3.6.7: /home/vlad/.local/share/virtualenvs/gtg-cnQmc6C0/bin/python3
• 3.6.7: /usr/bin/python3.6
• 3.6.7: /usr/bin/python3.6m
• 3.5.3: /home/vlad/.local/bin/pypy3
• 2.7.15rc1: /usr/bin/python

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.7',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-47-generic',
 'platform_system': 'Linux',
 'platform_version': '#50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019',
 'python_full_version': '3.6.7',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• LS_COLORS
• INSIDE_NEMO_PYTHON
• LC_MEASUREMENT
• LESSCLOSE
• LC_PAPER
• LC_MONETARY
• KIGITHUB
• LANG
• GDM_LANG
• DISPLAY
• GTK_OVERLAY_SCROLLING
• COLORTERM
• XDG_VTNR
• KICAD_PATH
• SSH_AUTH_SOCK
• VIRTUAL_ENV
• MANDATORY_PATH
• LC_NAME
• XDG_SESSION_ID
• XDG_GREETER_DATA_DIR
• USER
• DESKTOP_SESSION
• PIP_PYTHON_PATH
• GNOME_TERMINAL_SCREEN
• DEFAULTS_PATH
• QT_QPA_PLATFORMTHEME
• PWD
• HOME
• SSH_AGENT_PID
• QT_ACCESSIBILITY
• XDG_SESSION_TYPE
• XDG_DATA_DIRS
• PIPENV_ACTIVE
• XDG_SESSION_DESKTOP
• LC_ADDRESS
• LC_NUMERIC
• GTK_MODULES
• VTE_VERSION
• SHELL
• TERM
• XDG_SEAT_PATH
• XDG_CURRENT_DESKTOP
• GPG_AGENT_INFO
• GNOME_TERMINAL_SERVICE
• PYTHONDONTWRITEBYTECODE
• SHLVL
• XDG_SEAT
• LANGUAGE
• LC_TELEPHONE
• GDMSESSION
• GNOME_DESKTOP_SESSION_ID
• LOGNAME
• PIP_DISABLE_PIP_VERSION_CHECK
• DBUS_SESSION_BUS_ADDRESS
• XDG_RUNTIME_DIR
• XAUTHORITY
• XDG_SESSION_PATH
• XDG_CONFIG_DIRS
• PATH
• LC_IDENTIFICATION
• PS1
• SESSION_MANAGER
• LESSOPEN
• _
• PIP_SHIMS_BASE_MODULE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_ACTIVE: 1

Debug–specific environment variables:

• PATH: /home/vlad/.local/share/virtualenvs/gtg-cnQmc6C0/bin:/home/vlad/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/vlad/.local/bin:/home/vlad/.local/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /home/vlad/Programs/organizing/gtg
• VIRTUAL_ENV: /home/vlad/.local/share/virtualenvs/gtg-cnQmc6C0

---

Contents of Pipfile ('/home/vlad/Programs/organizing/gtg/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
nose = "*"
spec = "*"
pyflakes = "*"
pep8 = "*"
mock = "*"
liblarch = {git = "https://github.com/getting-things-gnome/liblarch.git",ref = "master"}

[requires]
python_version = "3.6"

[scripts]
dev = './gtg.sh'

Contents of Pipfile.lock ('/home/vlad/Programs/organizing/gtg/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "1b305c19130f0cd80840f0090f3dcd67442c476c71f5bf53e27fc7a35fe4c0d2"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "liblarch": {
            "git": "https://github.com/getting-things-gnome/liblarch.git",
            "ref": "ebd05dde26ce8dce2ab181d78bab64750e84a04c"
        },
        "mock": {
            "hashes": [
                "sha256:5ce3c71c5545b472da17b72268978914d0252980348636840bd34a00b5cc96c1",
                "sha256:b158b6df76edd239b8208d481dc46b6afd45a846b7812ff0ce58971cf5bc8bba"
            ],
            "index": "pypi",
            "version": "==2.0.0"
        },
        "nose": {
            "hashes": [
                "sha256:9ff7c6cc443f8c51994b34a667bbcf45afd6d945be7477b52e97516fd17c53ac",
                "sha256:dadcddc0aefbf99eea214e0f1232b94f2fa9bd98fa8353711dacb112bfcbbb2a",
                "sha256:f1bffef9cbc82628f6e7d7b40d7e255aefaa1adb6a1b1d26c69a8b79e6208a98"
            ],
            "index": "pypi",
            "version": "==1.3.7"
        },
        "pbr": {
            "hashes": [
                "sha256:6901995b9b686cb90cceba67a0f6d4d14ae003cd59bc12beb61549bdfbe3bc89",
                "sha256:d950c64aeea5456bbd147468382a5bb77fe692c13c9f00f0219814ce5b642755"
            ],
            "version": "==5.2.0"
        },
        "pep8": {
            "hashes": [
                "sha256:b22cfae5db09833bb9bd7c8463b53e1a9c9b39f12e304a8d0bba729c501827ee",
                "sha256:fe249b52e20498e59e0b5c5256aa52ee99fc295b26ec9eaa85776ffdb9fe6374"
            ],
            "index": "pypi",
            "version": "==1.7.1"
        },
        "pyflakes": {
            "hashes": [
                "sha256:17dbeb2e3f4d772725c777fabc446d5634d1038f234e77343108ce445ea69ce0",
                "sha256:d976835886f8c5b31d47970ed689944a0262b5f3afa00a5a7b4dc81e5449f8a2"
            ],
            "index": "pypi",
            "version": "==2.1.1"
        },
        "six": {
            "hashes": [
                "sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
                "sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
            ],
            "version": "==1.12.0"
        },
        "spec": {
            "hashes": [
                "sha256:7f805563ff1d11b449d419bc851b58859401e6bde183b21250f4fa21dd216545",
                "sha256:d50e3f6e8c9771c68c9b56151533ec46f1e0bfcc94a121a835175cb034bf6b31"
            ],
            "index": "pypi",
            "version": "==1.4.1"
        }
    },
    "develop": {}
}

P.S. also add pipenv --site-packages install output.
Now It's python3.6.7 instead of python3.6.6 in my system, but bug is still present.

pipenv --site-packages install

pipenv --site-packages install Creating a virtualenv for this project… Pipfile: /home/vlad/Desktop/temp/testing/test_venv/Pipfile Using /usr/bin/python3.6 (3.6.7) to create virtualenv… ⠋ Creating virtual environment...Using base prefix '/usr' New python executable in /home/vlad/.local/share/virtualenvs/test_venv-Omu8UKuC/bin/python3.6 Also creating executable in /home/vlad/.local/share/virtualenvs/test_venv-Omu8UKuC/bin/python Installing setuptools, pip, wheel...done. Running virtualenv with interpreter /usr/bin/python3.6

✔ Successfully created virtual environment!
Virtualenv location: /home/vlad/.local/share/virtualenvs/test_venv-Omu8UKuC
Installing dependencies from Pipfile.lock (8c4c43)…
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 10/10 — 00:00:15
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

[jxltom]

Which python path are you using to create virtualenv by pipenv --site-packages install? Could you output the content such as Using /Users/username/.pyenv/versions/3.6.6/bin/python3.6 (3.6.6) to create virtualenv...?

[Marviel]

I had this same issue some months ago and my only strategy was to pin an old pipenv version. I really hope this can be fixed soon, because the new versions of Pipenv have a lot to offer :/

#3504

^^^ That issue includes a dockerized reproducibility case.

[vlad0337187]

@jxltom , I used python3.6 for that (the only version except 2.7 in my system).
Added info you asked to my first message (to the end).

Checked, bug is still present.

[jxltom]

Thanks for the additional info!

[techalchemy]

Yeah we are consciously disabling user site access although I'm not sure why that is carrying into virtualenvs... I can't actually remember the reason why we needed to disable user site permission, but I believe it involved something about setuptools and editable installs...

[vlad0337187]

For me it should help accessing packages, which can't be installed in virtualenv.
For example, when developing GTK cross-distributive apps.
I always use system libraries, just make Python code working with them all.

Now what I did - I found virtualenv created by pipenv, made many soft links to directories for system packages, which is too long way - I should repeat it for every distributive I test app.

So, hope, it could be fixed.
Accessing to system packages, while also installed in virtualenv - helps to work in a more comfortable way.
Thanks.

[frostming]

Can you try against the master branch to confirm if it still exists?

[scresante]

@frostming I just tried with master and the issue is resolved for me.

Any idea when the next release will be?