Skip to content
This repository has been archived by the owner on Nov 10, 2023. It is now read-only.

More pex fixes for windows. #614

Closed
wants to merge 1 commit into from
Closed

More pex fixes for windows. #614

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions bin/buck.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
@echo off
set PYTHONPATH=%~dp0..\third-party\nailgun
python %~dp0..\programs\buck.py %*
13 changes: 2 additions & 11 deletions third-party/py/pex/pex/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,17 +269,8 @@ def link(self, src, dst, label=None):
self._ensure_parent(dst)
abs_src = src
abs_dst = os.path.join(self.chroot, dst)
try:
os.link(abs_src, abs_dst)
except OSError as e:
if e.errno == errno.EEXIST:
# File already exists, skip XXX -- ensure target and dest are same?
pass
elif e.errno == errno.EXDEV:
# Hard link across devices, fall back on copying
shutil.copyfile(abs_src, abs_dst)
else:
raise
safe_copy(abs_src, abs_dst, overwrite=False)
# If the file already exists, skip XXX -- ensure target and dest are same?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can actually avoid this path entirely if we just modify the code in src/com/facebook/buck/python/make_pex.py to prefer to copy on Windows. I had originally wrote something like this, but @andrewjcg suggested changing Buck's code so it was fewer changes to PEX itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually going to try to upstream these in pex-tool/pex#198

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You rock!


def write(self, data, dst, label=None, mode='wb'):
"""Write data to ``chroot/dst`` with optional label.
Expand Down
15 changes: 14 additions & 1 deletion third-party/py/pex/pex/compiler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

from __future__ import absolute_import

import contextlib
import os
import subprocess
import tempfile

Expand Down Expand Up @@ -58,6 +60,17 @@ def main(root, relpaths):
"""


@contextlib.contextmanager
def named_temporary_file():
fp, path = tempfile.mkstemp()
os.close(fp)
try:
with open(path, 'w') as fp:
yield fp
finally:
os.remove(path)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have to try this when I get into the office, but when I was using something like, this, I was getting an exception with the call to remove because the subprocess call hadn't finished closing it yet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hit that too, but it wasn't the subprocess call - it was the fp returned by mkstemp



class Compiler(object):
class Error(Exception):
"""Indicates an error compiling one or more python source files."""
Expand All @@ -78,7 +91,7 @@ def compile(self, root, relpaths):
:returns: A list of relative paths of the compiled bytecode files.
:raises: A :class:`Compiler.Error` if there was a problem bytecode compiling any of the files.
"""
with tempfile.NamedTemporaryFile() as fp:
with named_temporary_file() as fp:
fp.write(to_bytes(_COMPILER_MAIN % {'root': root, 'relpaths': relpaths}, encoding='utf-8'))
fp.flush()
process = subprocess.Popen([self._interpreter.binary, fp.name],
Expand Down
3 changes: 3 additions & 0 deletions third-party/py/pex/pex/pex.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,9 @@ def _site_libs(cls):
site_libs = set()
site_libs.update([sysconfig.get_python_lib(plat_specific=False),
sysconfig.get_python_lib(plat_specific=True)])
# On windows getsitepackages() returns the python stdlib too.
if sys.prefix in site_libs:
site_libs.remove(sys.prefix)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good find. I wasn't familiar with this issues, and I'm curious how you came across it; care to elaborate here (just on Github; the code is fine as-is).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the crash you saw was an extreme example of the bug - the windows box I used to test doesn't have any packages installed, so the actual thing I saw was an ImportError trying to import json. To trace it down further I set PEX_VERBOSE=5 and saw that the stdlib was being "scrubbed" from the path.

real_site_libs = set(os.path.realpath(path) for path in site_libs)
return site_libs | real_site_libs

Expand Down