Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

import path leak for six with OSX default python #302

Closed
kwlzn opened this issue Sep 28, 2016 · 13 comments
Closed

import path leak for six with OSX default python #302

kwlzn opened this issue Sep 28, 2016 · 13 comments
Assignees

Comments

@kwlzn
Copy link
Contributor

kwlzn commented Sep 28, 2016

when running a pex that bundles six with the OSX system python (/usr/bin/python), the import of six references the system path (an ancient version of six) rather than the one provided with the pex. this seems to relate to OSX's bespoke usage of the "Extras" path in their installation, which pex does not classify as user-site or site-packages during the sys.path scrubbing.

repro:

[illuminati source (master)]$ PATH=/usr/bin ~/bin/pex six -o /tmp/throwaway.pex
[illuminati source (master)]$ PATH=/usr/bin /tmp/throwaway.pex
Python 2.7.10 (default, Jul 30 2016, 18:31:42)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys, six
>>> print(sys.executable, six.__file__)
('/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc')
>>> from pprint import pprint
>>> pprint(sys.path)
['/tmp/throwaway.pex/.bootstrap',
 '/tmp/throwaway.pex',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 u'/Users/kwilson/.pex/install/six-1.10.0-py2.py3-none-any.whl.a99dfb27e60da3957f6667853b91ea52e173da0a/six-1.10.0-py2.py3-none-any.whl']
@kwlzn
Copy link
Contributor Author

kwlzn commented Sep 28, 2016

related: pypa/pip#2468

@lorencarvalho
Copy link
Contributor

how do you intend to address this? the first thing that comes to my mind is just scrubbing 'Extras' from sys.path, though that could get messy...

@kwlzn
Copy link
Contributor Author

kwlzn commented Oct 3, 2016

haven't dug into it but on the surface it seems like it could be as simple as just scrubbing os.path.join(sys.prefix, 'Extras') if present, I think.

let me know if you have a better idea.

@lorencarvalho
Copy link
Contributor

I think that's pretty reasonable, considering the scrubbing that pex already does.

@GregDracoulis
Copy link

Is there any interim solution to this for affected users?

@kwlzn
Copy link
Contributor Author

kwlzn commented Dec 8, 2016

this is limited to the OSX default interpreter, so one possible solution would be to use an alternate python interpreter (e.g. one from python.org, homebrew or built from sources).

@GregDracoulis
Copy link

I'm running into this issue even when using a virtualenv with either a python.org interpreter or homebrew interpreter. Is there by chance any way to tell Pants which interpreter to use with PEX? I can't find anything in the documentation, but I'm new to working with this so I might be missing something.

Or is there some way maybe to specify additional paths to scrub? I could see that being fairly useful if it exists. We have a couple devs running into this issue and it'd be great if we didn't have to wipe and revert back to El Capitan to get our local builds working.

@kwlzn
Copy link
Contributor Author

kwlzn commented Jan 3, 2017

can you demonstrate the issue with a python.org or homebrew interpreter? I don't seem to reproduce that on my end and afaict this is very specific to modern OSX /usr/bin/python.

for posterity, here's a quick code-based workaround that was discussed with another user:

$ /usr/bin/python ~/bin/pex six
Python 2.7.10 (default, Jul 30 2016, 19:40:32) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys
>>> sys.path = [p for p in sys.path if 'Extras/lib/python' not in p]
>>> import six; six.__file__
'/private/var/folders/3t/xkwqrkld4xxgklk2s4n41jb80000gn/T/tmpEXfjuN/.deps/six-1.10.0-py2.py3-none-any.whl/six.py'
>>>

@mateor
Copy link

mateor commented Jan 23, 2017

mateo-2:pants mateo$ ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
<bootstraps>
                     tests/python/pants_test/util/test_contextutil.py:16: in <module>
                         import mock
                     .pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/mock-1.3.0-py2.py3-none-any.whl/mock/__init__.py:2: in <module>
                         import mock.mock as _mock
                     .pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/mock-1.3.0-py2.py3-none-any.whl/mock/mock.py:68: in <module>
                         from six import wraps
                     E   ImportError: cannot import name wraps
                     ============ 1 error in 0.06 seconds =============

mateo-2:~ mateo$ /usr/local/bin/python2.7
Python 2.7.13 (default, Jan 17 2017, 19:17:41)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

I was able to repro as you see above - this was triggered when setting up Pants on a new MacBookPro. It was using the homebrew python and had a reboot mixed in, on a fresh clone of Pants at today's HEAD.

The TLDR is that, on a new MacBookPro at least, your set of scrubbed sites is perhaps incomplete.

mateo-2:pants mateo$ cat /Library/Python/2.7/site-packages/Extras.pth
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
mateo-2:pants mateo$ PEX_VERBOSE=4 ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
<snip>
pex: Not a tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
<snip>
mateo-2:~ mateo$ /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Python 2.7.10 (default, Jul 30 2016, 19:40:32)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import six
>>> from six import wraps
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name wraps

But I don't see why Pex is constructing the path the way it does? It seems like the .deps should be above the randomly found site-packages dirs on disk:

mateo-2:pants mateo$ PEX_VERBOSE=4 ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
<snip>
pex: PYTHONPATH contains:
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.bootstrap
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4
                     pex:     /Users/mateo/dev/pants/src/python
                     pex:     /usr/local/lib/python2.7/site-packages
                     pex:   * /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
                     pex:   * /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/linecache2-1.0.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pytest_cov-1.8.1-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pytest-2.6.4-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/cov_core-1.15.0-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/funcsigs-1.0.2-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/six-1.10.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pytest_timeout-0.5-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/py-1.4.32-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/traceback2-1.4.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/unittest2-1.1.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/argparse-1.4.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pbr-1.10.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/coverage-3.7.1-cp27-cp27m-macosx_10_12_intel.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/ansicolors-1.0.2-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/mock-1.3.0-py2.py3-none-any.whl

This passes for me - unless I uninstall the six module from pip - then it falls back to the System 1.4.1.

Workaround for the afflicted is to install homebrew python, install six or another problem module, and then override the global PYTHONPATH, a la

export PYTHONPATH="/usr/local/lib/python2.7/site-packages/"

Now - if you do that and then uninstall the global six...

mateo-2:pants mateo$ pip uninstall six
Uninstalling six-1.10.0:
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/DESCRIPTION.rst
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/INSTALLER
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/METADATA
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/RECORD
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/WHEEL
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/metadata.json
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/top_level.txt
  /usr/local/lib/python2.7/site-packages/six.py
  /usr/local/lib/python2.7/site-packages/six.pyc
Proceed (y/n)? y
  Successfully uninstalled six-1.10.0
mateo-2:pants mateo$ PEX_VERBOSE=4 ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
Traceback (most recent call last):
  File "/Users/mateo/dev/pants/src/python/pants/bin/pants_exe.py", line 18, in <module>
    from pants.bin.pants_runner import PantsRunner  # isort:skip
  File "/Users/mateo/dev/pants/src/python/pants/bin/pants_runner.py", line 12, in <module>
    from pants.bin.remote_pants_runner import RemotePantsRunner
  File "/Users/mateo/dev/pants/src/python/pants/bin/remote_pants_runner.py", line 12, in <module>
    from pants.java.nailgun_client import NailgunClient
  File "/Users/mateo/dev/pants/src/python/pants/java/nailgun_client.py", line 14, in <module>
    from pants.java.nailgun_io import NailgunStreamReader
  File "/Users/mateo/dev/pants/src/python/pants/java/nailgun_io.py", line 16, in <module>
    from pants.java.nailgun_protocol import ChunkType, NailgunProtocol
  File "/Users/mateo/dev/pants/src/python/pants/java/nailgun_protocol.py", line 10, in <module>
    import six
ImportError: No module named six
mateo-2:pants mateo$

Perhaps this is all expected.

I had a couple false starts to this report where I thought I was seeing non-determinism in the sys.path patching - it seemed to do surprising things to the order, and perhaps reverse the ordering at times? It is completely possible that I was just getting fooled by the cache hits.

But I can provide more of the verbose output or sys.paths as needed.

@benjyw
Copy link
Collaborator

benjyw commented Jan 29, 2017

Any updates on this? I'm hitting it too now. I have a brew python installed, but Pants selects the system one for whatever reason, and the ancient version of six there is leaking into the chroot.

@mateor
Copy link

mateor commented Jan 29, 2017 via email

@kwlzn kwlzn self-assigned this Mar 6, 2017
@kwlzn
Copy link
Contributor Author

kwlzn commented Mar 6, 2017

grabbing this one due to user impact.

@kwlzn
Copy link
Contributor Author

kwlzn commented Mar 7, 2017

reviewable here: #370

@kwlzn kwlzn closed this as completed in #370 Mar 7, 2017
kwlzn added a commit that referenced this issue Mar 7, 2017
Fixes #302

before:

    [omerta pex (kwlzn/osxtras)]$ pex --version
    pex 1.2.3
    [omerta pex (kwlzn/osxtras)]$ pex six -o /tmp/six_broke.pex
    [omerta pex (kwlzn/osxtras)]$ PEX_VERBOSE=9 PEX_PYTHON=/usr/bin/python /tmp/six_broke.pex 
    pex: Detected PEX_PYTHON, re-exec to /usr/bin/python
    ...
    Python 2.7.10 (default, Jul 30 2016, 19:40:32) 
    [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    (InteractiveConsole)
    >>> from six import wraps
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
    ImportError: cannot import name wraps
    >>> import six; six.__file__
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc'
    >>>

after:

    [omerta pex (kwlzn/osxtras)]$ tox -e py27-package
    ...
    [omerta pex (kwlzn/osxtras)]$ ./dist/pex27 six -o /tmp/six.pex
    ...
    [omerta pex (kwlzn/osxtras)]$ PEX_VERBOSE=9 PEX_PYTHON=/usr/bin/python /tmp/six.pex 
    pex: Detected PEX_PYTHON, re-exec to /usr/bin/python
    ...
    pex: Found .pth file: /Library/Python/2.7/site-packages/Extras.pth
    pex: Found site extra: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
    pex: Found site extra: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
    ...
    pex: Tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
    pex: Tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
    ...
    pex: Scrubbing from site-packages: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
    pex: Scrubbing from site-packages: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
    ...
    Python 2.7.10 (default, Jul 30 2016, 19:40:32) 
    [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    (InteractiveConsole)
    >>> from six import wraps
    >>> import six; six.__file__
    '/Users/kwilson/.pex/install/six-1.10.0-py2.py3-none-any.whl.a99dfb27e60da3957f6667853b91ea52e173da0a/six-1.10.0-py2.py3-none-any.whl/six.pyc'
    >>>
butlern pushed a commit to butlern/pex that referenced this issue Jun 30, 2017
Fixes pex-tool#302

before:

    [omerta pex (kwlzn/osxtras)]$ pex --version
    pex 1.2.3
    [omerta pex (kwlzn/osxtras)]$ pex six -o /tmp/six_broke.pex
    [omerta pex (kwlzn/osxtras)]$ PEX_VERBOSE=9 PEX_PYTHON=/usr/bin/python /tmp/six_broke.pex 
    pex: Detected PEX_PYTHON, re-exec to /usr/bin/python
    ...
    Python 2.7.10 (default, Jul 30 2016, 19:40:32) 
    [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    (InteractiveConsole)
    >>> from six import wraps
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
    ImportError: cannot import name wraps
    >>> import six; six.__file__
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc'
    >>>

after:

    [omerta pex (kwlzn/osxtras)]$ tox -e py27-package
    ...
    [omerta pex (kwlzn/osxtras)]$ ./dist/pex27 six -o /tmp/six.pex
    ...
    [omerta pex (kwlzn/osxtras)]$ PEX_VERBOSE=9 PEX_PYTHON=/usr/bin/python /tmp/six.pex 
    pex: Detected PEX_PYTHON, re-exec to /usr/bin/python
    ...
    pex: Found .pth file: /Library/Python/2.7/site-packages/Extras.pth
    pex: Found site extra: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
    pex: Found site extra: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
    ...
    pex: Tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
    pex: Tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
    ...
    pex: Scrubbing from site-packages: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
    pex: Scrubbing from site-packages: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
    ...
    Python 2.7.10 (default, Jul 30 2016, 19:40:32) 
    [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    (InteractiveConsole)
    >>> from six import wraps
    >>> import six; six.__file__
    '/Users/kwilson/.pex/install/six-1.10.0-py2.py3-none-any.whl.a99dfb27e60da3957f6667853b91ea52e173da0a/six-1.10.0-py2.py3-none-any.whl/six.pyc'
    >>>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants