import path leak for `six` with OSX default python #302

kwlzn · 2016-09-28T21:42:39Z

when running a pex that bundles six with the OSX system python (/usr/bin/python), the import of six references the system path (an ancient version of six) rather than the one provided with the pex. this seems to relate to OSX's bespoke usage of the "Extras" path in their installation, which pex does not classify as user-site or site-packages during the sys.path scrubbing.

repro:

[illuminati source (master)]$ PATH=/usr/bin ~/bin/pex six -o /tmp/throwaway.pex
[illuminati source (master)]$ PATH=/usr/bin /tmp/throwaway.pex
Python 2.7.10 (default, Jul 30 2016, 18:31:42)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys, six
>>> print(sys.executable, six.__file__)
('/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc')
>>> from pprint import pprint
>>> pprint(sys.path)
['/tmp/throwaway.pex/.bootstrap',
 '/tmp/throwaway.pex',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 u'/Users/kwilson/.pex/install/six-1.10.0-py2.py3-none-any.whl.a99dfb27e60da3957f6667853b91ea52e173da0a/six-1.10.0-py2.py3-none-any.whl']

The text was updated successfully, but these errors were encountered:

kwlzn · 2016-09-28T22:14:26Z

related: pypa/pip#2468

lorencarvalho · 2016-10-01T04:30:47Z

how do you intend to address this? the first thing that comes to my mind is just scrubbing 'Extras' from sys.path, though that could get messy...

kwlzn · 2016-10-03T17:35:35Z

haven't dug into it but on the surface it seems like it could be as simple as just scrubbing os.path.join(sys.prefix, 'Extras') if present, I think.

let me know if you have a better idea.

lorencarvalho · 2016-10-06T19:36:13Z

I think that's pretty reasonable, considering the scrubbing that pex already does.

GregDracoulis · 2016-12-08T21:30:53Z

Is there any interim solution to this for affected users?

kwlzn · 2016-12-08T22:47:43Z

this is limited to the OSX default interpreter, so one possible solution would be to use an alternate python interpreter (e.g. one from python.org, homebrew or built from sources).

GregDracoulis · 2016-12-09T15:27:03Z

I'm running into this issue even when using a virtualenv with either a python.org interpreter or homebrew interpreter. Is there by chance any way to tell Pants which interpreter to use with PEX? I can't find anything in the documentation, but I'm new to working with this so I might be missing something.

Or is there some way maybe to specify additional paths to scrub? I could see that being fairly useful if it exists. We have a couple devs running into this issue and it'd be great if we didn't have to wipe and revert back to El Capitan to get our local builds working.

kwlzn · 2017-01-03T23:50:32Z

can you demonstrate the issue with a python.org or homebrew interpreter? I don't seem to reproduce that on my end and afaict this is very specific to modern OSX /usr/bin/python.

for posterity, here's a quick code-based workaround that was discussed with another user:

$ /usr/bin/python ~/bin/pex six
Python 2.7.10 (default, Jul 30 2016, 19:40:32) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys
>>> sys.path = [p for p in sys.path if 'Extras/lib/python' not in p]
>>> import six; six.__file__
'/private/var/folders/3t/xkwqrkld4xxgklk2s4n41jb80000gn/T/tmpEXfjuN/.deps/six-1.10.0-py2.py3-none-any.whl/six.py'
>>>

mateor · 2017-01-23T08:17:36Z

mateo-2:pants mateo$ ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
<bootstraps>
                     tests/python/pants_test/util/test_contextutil.py:16: in <module>
                         import mock
                     .pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/mock-1.3.0-py2.py3-none-any.whl/mock/__init__.py:2: in <module>
                         import mock.mock as _mock
                     .pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/mock-1.3.0-py2.py3-none-any.whl/mock/mock.py:68: in <module>
                         from six import wraps
                     E   ImportError: cannot import name wraps
                     ============ 1 error in 0.06 seconds =============

mateo-2:~ mateo$ /usr/local/bin/python2.7
Python 2.7.13 (default, Jan 17 2017, 19:17:41)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

I was able to repro as you see above - this was triggered when setting up Pants on a new MacBookPro. It was using the homebrew python and had a reboot mixed in, on a fresh clone of Pants at today's HEAD.

The TLDR is that, on a new MacBookPro at least, your set of scrubbed sites is perhaps incomplete.

mateo-2:pants mateo$ cat /Library/Python/2.7/site-packages/Extras.pth
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
mateo-2:pants mateo$ PEX_VERBOSE=4 ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
<snip>
pex: Not a tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
<snip>
mateo-2:~ mateo$ /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Python 2.7.10 (default, Jul 30 2016, 19:40:32)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import six
>>> from six import wraps
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name wraps

But I don't see why Pex is constructing the path the way it does? It seems like the .deps should be above the randomly found site-packages dirs on disk:

mateo-2:pants mateo$ PEX_VERBOSE=4 ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
<snip>
pex: PYTHONPATH contains:
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.bootstrap
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4
                     pex:     /Users/mateo/dev/pants/src/python
                     pex:     /usr/local/lib/python2.7/site-packages
                     pex:   * /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
                     pex:   * /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/linecache2-1.0.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pytest_cov-1.8.1-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pytest-2.6.4-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/cov_core-1.15.0-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/funcsigs-1.0.2-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/six-1.10.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pytest_timeout-0.5-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/py-1.4.32-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/traceback2-1.4.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/unittest2-1.1.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/argparse-1.4.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pbr-1.10.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/coverage-3.7.1-cp27-cp27m-macosx_10_12_intel.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/ansicolors-1.0.2-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/mock-1.3.0-py2.py3-none-any.whl

This passes for me - unless I uninstall the six module from pip - then it falls back to the System 1.4.1.

Workaround for the afflicted is to install homebrew python, install six or another problem module, and then override the global PYTHONPATH, a la

export PYTHONPATH="/usr/local/lib/python2.7/site-packages/"

Now - if you do that and then uninstall the global six...

mateo-2:pants mateo$ pip uninstall six
Uninstalling six-1.10.0:
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/DESCRIPTION.rst
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/INSTALLER
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/METADATA
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/RECORD
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/WHEEL
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/metadata.json
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/top_level.txt
  /usr/local/lib/python2.7/site-packages/six.py
  /usr/local/lib/python2.7/site-packages/six.pyc
Proceed (y/n)? y
  Successfully uninstalled six-1.10.0
mateo-2:pants mateo$ PEX_VERBOSE=4 ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
Traceback (most recent call last):
  File "/Users/mateo/dev/pants/src/python/pants/bin/pants_exe.py", line 18, in <module>
    from pants.bin.pants_runner import PantsRunner  # isort:skip
  File "/Users/mateo/dev/pants/src/python/pants/bin/pants_runner.py", line 12, in <module>
    from pants.bin.remote_pants_runner import RemotePantsRunner
  File "/Users/mateo/dev/pants/src/python/pants/bin/remote_pants_runner.py", line 12, in <module>
    from pants.java.nailgun_client import NailgunClient
  File "/Users/mateo/dev/pants/src/python/pants/java/nailgun_client.py", line 14, in <module>
    from pants.java.nailgun_io import NailgunStreamReader
  File "/Users/mateo/dev/pants/src/python/pants/java/nailgun_io.py", line 16, in <module>
    from pants.java.nailgun_protocol import ChunkType, NailgunProtocol
  File "/Users/mateo/dev/pants/src/python/pants/java/nailgun_protocol.py", line 10, in <module>
    import six
ImportError: No module named six
mateo-2:pants mateo$

Perhaps this is all expected.

I had a couple false starts to this report where I thought I was seeing non-determinism in the sys.path patching - it seemed to do surprising things to the order, and perhaps reverse the ordering at times? It is completely possible that I was just getting fooled by the cache hits.

But I can provide more of the verbose output or sys.paths as needed.

benjyw · 2017-01-29T08:11:32Z

Any updates on this? I'm hitting it too now. I have a brew python installed, but Pants selects the system one for whatever reason, and the ancient version of six there is leaking into the chroot.

mateor · 2017-01-29T18:09:36Z

It was probably buried in the breadth of my reply but you can hardcode the PYTHONPATH to point to just the homebrew one and at least proceed.

…

On Sun, Jan 29, 2017, 3:11 AM Benjy Weinberger ***@***.***> wrote: Any updates on this? I'm hitting it too now. I have a brew python installed, but Pants selects the system one for whatever reason, and the ancient version of six there is leaking into the chroot. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#302 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABqcAadjDjPg7MalJ4lszwoXbo9uctJFks5rXEm0gaJpZM4KJWTZ> .

kwlzn · 2017-03-06T21:22:33Z

grabbing this one due to user impact.

kwlzn · 2017-03-07T04:27:02Z

reviewable here: #370

Fixes #302 before: [omerta pex (kwlzn/osxtras)]$ pex --version pex 1.2.3 [omerta pex (kwlzn/osxtras)]$ pex six -o /tmp/six_broke.pex [omerta pex (kwlzn/osxtras)]$ PEX_VERBOSE=9 PEX_PYTHON=/usr/bin/python /tmp/six_broke.pex pex: Detected PEX_PYTHON, re-exec to /usr/bin/python ... Python 2.7.10 (default, Jul 30 2016, 19:40:32) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> from six import wraps Traceback (most recent call last): File "<console>", line 1, in <module> ImportError: cannot import name wraps >>> import six; six.__file__ '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc' >>> after: [omerta pex (kwlzn/osxtras)]$ tox -e py27-package ... [omerta pex (kwlzn/osxtras)]$ ./dist/pex27 six -o /tmp/six.pex ... [omerta pex (kwlzn/osxtras)]$ PEX_VERBOSE=9 PEX_PYTHON=/usr/bin/python /tmp/six.pex pex: Detected PEX_PYTHON, re-exec to /usr/bin/python ... pex: Found .pth file: /Library/Python/2.7/site-packages/Extras.pth pex: Found site extra: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python pex: Found site extra: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC ... pex: Tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python pex: Tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC ... pex: Scrubbing from site-packages: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python pex: Scrubbing from site-packages: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC ... Python 2.7.10 (default, Jul 30 2016, 19:40:32) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> from six import wraps >>> import six; six.__file__ '/Users/kwilson/.pex/install/six-1.10.0-py2.py3-none-any.whl.a99dfb27e60da3957f6667853b91ea52e173da0a/six-1.10.0-py2.py3-none-any.whl/six.pyc' >>>

Fixes pex-tool#302 before: [omerta pex (kwlzn/osxtras)]$ pex --version pex 1.2.3 [omerta pex (kwlzn/osxtras)]$ pex six -o /tmp/six_broke.pex [omerta pex (kwlzn/osxtras)]$ PEX_VERBOSE=9 PEX_PYTHON=/usr/bin/python /tmp/six_broke.pex pex: Detected PEX_PYTHON, re-exec to /usr/bin/python ... Python 2.7.10 (default, Jul 30 2016, 19:40:32) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> from six import wraps Traceback (most recent call last): File "<console>", line 1, in <module> ImportError: cannot import name wraps >>> import six; six.__file__ '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc' >>> after: [omerta pex (kwlzn/osxtras)]$ tox -e py27-package ... [omerta pex (kwlzn/osxtras)]$ ./dist/pex27 six -o /tmp/six.pex ... [omerta pex (kwlzn/osxtras)]$ PEX_VERBOSE=9 PEX_PYTHON=/usr/bin/python /tmp/six.pex pex: Detected PEX_PYTHON, re-exec to /usr/bin/python ... pex: Found .pth file: /Library/Python/2.7/site-packages/Extras.pth pex: Found site extra: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python pex: Found site extra: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC ... pex: Tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python pex: Tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC ... pex: Scrubbing from site-packages: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python pex: Scrubbing from site-packages: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC ... Python 2.7.10 (default, Jul 30 2016, 19:40:32) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> from six import wraps >>> import six; six.__file__ '/Users/kwilson/.pex/install/six-1.10.0-py2.py3-none-any.whl.a99dfb27e60da3957f6667853b91ea52e173da0a/six-1.10.0-py2.py3-none-any.whl/six.pyc' >>>

kwlzn mentioned this issue Nov 1, 2016

sys.path isn't sane by default pantsbuild/setup#16

Closed

JustinVenus mentioned this issue Nov 2, 2016

scrub Extras from sys.path on macOS Sierra #312

Closed

kwlzn self-assigned this Mar 6, 2017

kwlzn mentioned this issue Mar 7, 2017

Treat .pth injected paths as extras. #370

Merged

kwlzn closed this as completed in #370 Mar 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

import path leak for `six` with OSX default python #302

import path leak for `six` with OSX default python #302

kwlzn commented Sep 28, 2016

kwlzn commented Sep 28, 2016

lorencarvalho commented Oct 1, 2016

kwlzn commented Oct 3, 2016

lorencarvalho commented Oct 6, 2016

GregDracoulis commented Dec 8, 2016

kwlzn commented Dec 8, 2016

GregDracoulis commented Dec 9, 2016

kwlzn commented Jan 3, 2017

mateor commented Jan 23, 2017

benjyw commented Jan 29, 2017

mateor commented Jan 29, 2017 via email

kwlzn commented Mar 6, 2017

kwlzn commented Mar 7, 2017

import path leak for six with OSX default python #302

import path leak for six with OSX default python #302

Comments

kwlzn commented Sep 28, 2016

kwlzn commented Sep 28, 2016

lorencarvalho commented Oct 1, 2016

kwlzn commented Oct 3, 2016

lorencarvalho commented Oct 6, 2016

GregDracoulis commented Dec 8, 2016

kwlzn commented Dec 8, 2016

GregDracoulis commented Dec 9, 2016

kwlzn commented Jan 3, 2017

mateor commented Jan 23, 2017

benjyw commented Jan 29, 2017

mateor commented Jan 29, 2017 via email

kwlzn commented Mar 6, 2017

kwlzn commented Mar 7, 2017

import path leak for `six` with OSX default python #302

import path leak for `six` with OSX default python #302