Python 2: SyntaxError when installing distributions with coding hints in setup.py

[saj]

I think this is a regression introduced in #231, which was released in pex 1.1.7. I cannot reproduce this problem on pex 1.1.6. Affects Python 2 only.

Reproduce with:

wget -q 'https://github.com/pantsbuild/pex/releases/download/v1.1.7/pex27'
chmod a+x pex27
./pex27 backoff

Stack trace:

**** Failed to install backoff-1.1.0. stdout:

**** Failed to install backoff-1.1.0. stderr:
Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "setup.py", line 0
SyntaxError: encoding declaration in Unicode string

Traceback (most recent call last):
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 328, in execute
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 260, in _wrap_coverage
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 292, in _wrap_profiling
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 371, in _execute
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 429, in execute_entry
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 447, in execute_pkg_resources
  File "/Users/saj/.pex/install/pex-1.1.7-py2.py3-none-any.whl.1089a1bbad10037ad29421de8a162a77c90c8133/pex-1.1.7-py2.py3-none-any.whl/pex/bin/pex.py", line 538, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/Users/saj/.pex/install/pex-1.1.7-py2.py3-none-any.whl.1089a1bbad10037ad29421de8a162a77c90c8133/pex-1.1.7-py2.py3-none-any.whl/pex/bin/pex.py", line 489, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/Users/saj/.pex/install/pex-1.1.7-py2.py3-none-any.whl.1089a1bbad10037ad29421de8a162a77c90c8133/pex-1.1.7-py2.py3-none-any.whl/pex/resolver.py", line 200, in resolve
    dist = self.build(package, resolvable.options)
  File "/Users/saj/.pex/install/pex-1.1.7-py2.py3-none-any.whl.1089a1bbad10037ad29421de8a162a77c90c8133/pex-1.1.7-py2.py3-none-any.whl/pex/resolver.py", line 257, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/Users/saj/.pex/install/pex-1.1.7-py2.py3-none-any.whl.1089a1bbad10037ad29421de8a162a77c90c8133/pex-1.1.7-py2.py3-none-any.whl/pex/resolver.py", line 168, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage(u'file:///Users/saj/.pex/build/backoff-1.1.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

Even if we were to ignore the problem above, I don't think it is right to assume a UTF-8 encoding on all files.

From PEP 263:

If a Unicode string with a coding declaration is passed to compile(), a SyntaxError will be raised.

From the same PEP:

The builtin compile() API will be enhanced to accept Unicode as input. 8-bit string input is subject to the standard procedure for encoding detection as described above.

Instead of opening setup.py as a text file and decoding it ourselves, it sounds like it should be possible to hand a binary stream directly to the compile() builtin and let Python deal with this muck.

[kwlzn]

seems reasonable to me - would like to get some basic test coverage added for this area tho to help prevent future issues.

[kwlzn]

I've cut a new release at https://pypi.python.org/pypi/pex/1.1.9 with this fix. if anyone feels like adding better test coverage around this area for various known problematic packages, would gladly accept a PR - or otherwise I'll plan to circle back here.

thanks again @saj !