Improve PEX_PATH handling in pex.py #421

CMLivingston · 2017-10-09T21:34:50Z

Replace all usages of self._vars.PEX_PATH with a unified pex path composed from merging pex_path in PEX-INFO and PEX_PATH set in the environment.

Problem:
pex_info.pex_path was being overwritten in _activate() in pex.py by the path set in the environment.

Solution:
Implement a util function to merge the paths into a single pex path for usage throughout pex.py, giving priority to the PEX-INFO pex path if both are set.

…ath property of PEX-INFO metadata. Fallback to PEX_PATH set in the environment if neccesary for compatability.

…ified pex path for PEXEnvironment activation; Add associated integration test.

kwlzn · 2017-10-10T22:38:00Z

pex/pex.py

      if pex_info.pex_path:
-        # set up other environments as specified in PEX_PATH
+        # set up other environments as specified in pex_path
        for pex_path in filter(None, pex_info.pex_path.split(os.pathsep)):


how about instead of assigning to temp vars and then joining + mutating pex_info.pex_path here, just combining these into a new temporary list and then handling that directly?

also, would it make more sense to use self._vars.PEX_PATH as the source of the env var vs self._pex_info_overrides.pex_path? that's how it worked prior to your last change.

e.g.

# N.B. `PEX_PATH` entries written into `PEX-INFO` take precedence over those set in the environment. combined_pex_path = self._merge_split(pex_info.pex_path, self._vars.PEX_PATH) if combined_pex_path: for pex_path in combined_pex_path: ...

kwlzn · 2017-10-10T22:42:06Z

pex/pex.py

+  def _merge_split(self, *paths):
+    merged_paths = ":".join(filter(None, paths))
+    return filter(None, merged_paths.split(':'))
+


could reduce this to a one liner list comprehension:

def _merge_split(self, *paths): return [p for p in ':'.join(paths).split(':') if p]

with the if p bit doing the same as the filter(None, ...) above.

kwlzn · 2017-10-10T22:46:09Z

pex/pex.py

@@ -68,11 +68,15 @@ def _activate(self):

      # set up the local .pex environment
      pex_info = self._pex_info.copy()
+      pex_info_pex_path = pex_info.pex_path
+      overrides_pex_path = self._pex_info_overrides.pex_path
      pex_info.update(self._pex_info_overrides)


I assume that you found that this pex_info.update() call does not actually combine env vars + PEX-INFO, but instead clobbers one or the other?

Yup! Env vars clobber the existing vars set by pex info.

kwlzn · 2017-10-10T23:04:43Z

tests/test_integration.py

+
+    # create a pex for environment PEX_PATH
+    pex3_path = os.path.join(output_dir, 'pex3.pex')
+    res3 = run_pex_command(['--disable-cache', 'sqlalchemy', '-o', pex3_path])


also, iirc sqlalchemy is a pretty heavy dep. how about something lighter, like bitarray, enum34, mox etc?

… party dep for integration test case

CMLivingston · 2017-10-11T17:55:16Z

pex/pex.py

      pex_info.update(self._pex_info_overrides)
+      pex_info.pex_path = ':'.join(filter(None, combined_pex_paths))


I figure it makes more sense to use a temp var here rather than construct a new pex info API to preserve the overrides where we want pex_info vars overwritten. Doing something like pex_info.merge_pex_paths_from_env(self._vars.PEX_PATH) wouldn't be possible within this context as it would require more temp vars/code to ensure that pex_info.pex_path comes first in the unified path.

pex_info.merge_pex_paths_from_env(self._vars.PEX_PATH) should totally work, and it would be a cleaner API here in particular because it doesn't involve you as a caller having to remember to put the content of pex_info.pex_path at the head before setting it - it would just merge it for you.

the implementation would look approximately like this:

from pex.util import merge_split class PexInfo(...): ... def merge_pex_path(self, pex_path): """Merges a new PEX_PATH definition into the existing one (if any). :param string pex_path: The PEX_PATH to merge. """ if not pex_path: return self.pex_path = ':'.join(merge_split(self.pex_path, pex_path))

bonus points if you provide a flag on merge_split to enable inline deduping.

kwlzn · 2017-10-11T17:44:26Z

pex/pex.py

@@ -132,7 +137,7 @@ def _get_site_packages():
  def site_libs(cls):
    site_libs = cls._get_site_packages()
    site_libs.update([sysconfig.get_python_lib(plat_specific=False),
-                      sysconfig.get_python_lib(plat_specific=True)])
+      sysconfig.get_python_lib(plat_specific=True)])


unintentional re-indent?

kwlzn · 2017-10-11T17:44:33Z

pex/pex.py

@@ -276,10 +281,10 @@ def patch_all(path, path_importer_cache, modules):
      patch_dict(sys.modules, modules)

    old_sys_path, old_sys_path_importer_cache, old_sys_modules = (
-        sys.path[:], sys.path_importer_cache.copy(), sys.modules.copy())
+      sys.path[:], sys.path_importer_cache.copy(), sys.modules.copy())


unintentional re-indent?

kwlzn · 2017-10-11T20:24:44Z

pex/pex.py

      pex_info.update(self._pex_info_overrides)
+      pex_info.pex_path = ':'.join(filter(None, combined_pex_paths))


pex_info.merge_pex_paths_from_env(self._vars.PEX_PATH) should totally work, and it would be a cleaner API here in particular because it doesn't involve you as a caller having to remember to put the content of pex_info.pex_path at the head before setting it - it would just merge it for you.

the implementation would look approximately like this:

from pex.util import merge_split class PexInfo(...): ... def merge_pex_path(self, pex_path): """Merges a new PEX_PATH definition into the existing one (if any). :param string pex_path: The PEX_PATH to merge. """ if not pex_path: return self.pex_path = ':'.join(merge_split(self.pex_path, pex_path))

bonus points if you provide a flag on merge_split to enable inline deduping.

…PATH from from_env method

kwlzn

lgtm! two tiny straggler requests.

kwlzn · 2017-10-11T22:23:29Z

pex/util.py

+
+def merge_split(*paths):
+  """
+  Merge paths into a single path delimited by colons and split on colons to return


the docstring should start on the opening line like:

"""Docstring starts here. ... """

otherwise it won't appear correctly in IDEs/help()/etc.

kwlzn · 2017-10-11T22:24:06Z

pex/util.py

+
+
+def merge_split(*paths):
+  """


a quick test for this in test_util.py would probably be good.

Chris Livingston added 4 commits October 9, 2017 14:26

Patch sys module with a new sys.path repsecting the newly added pex_p…

6c5ed74

…ath property of PEX-INFO metadata. Fallback to PEX_PATH set in the environment if neccesary for compatability.

Add merging of environment pex path and pex info pex path into one un…

3a563f4

…ified pex path for PEXEnvironment activation; Add associated integration test.

Change 3rd party library used in integration test

999c6bf

Add pypy compatible 3rd party module for integration test

19fb285

kwlzn reviewed Oct 10, 2017

View reviewed changes

Clean up helper function and pex path merging, add lighter-weight 3rd…

4db4ea4

… party dep for integration test case

kwlzn mentioned this pull request Oct 11, 2017

Release 1.2.13 #425

Closed

Cleaner composition of joined pex path variable where possible

fbbaf91

CMLivingston commented Oct 11, 2017

View reviewed changes

kwlzn reviewed Oct 11, 2017

View reviewed changes

New api for PexInfo object: merge pex path into pex path; remove PEX_…

aed5211

…PATH from from_env method

kwlzn approved these changes Oct 11, 2017

View reviewed changes

CMLivingston changed the title ~~Patch sys module using pex_path from PEX-INFO metadata~~ Improve PEX_PATH handling in pex.py Oct 11, 2017

Chris Livingston added 3 commits October 11, 2017 15:44

Add test for new util function

df5f4f4

Add two more test cases

c663b40

isort changes

69bb04a

kwlzn merged commit 39bcfaa into pex-tool:master Oct 12, 2017

jsirois mentioned this pull request Oct 1, 2021

Pants 2.7.0: ./pants repl fails when any 3rdparty deps are required. pantsbuild/pants#13075

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve PEX_PATH handling in pex.py #421

Improve PEX_PATH handling in pex.py #421

CMLivingston commented Oct 9, 2017 •

edited

Loading

kwlzn Oct 10, 2017

kwlzn Oct 10, 2017

kwlzn Oct 10, 2017

CMLivingston Oct 10, 2017

kwlzn Oct 10, 2017

CMLivingston Oct 11, 2017 •

edited

Loading

kwlzn Oct 11, 2017

kwlzn Oct 11, 2017

kwlzn Oct 11, 2017

kwlzn Oct 11, 2017

kwlzn left a comment

kwlzn Oct 11, 2017

kwlzn Oct 11, 2017

		pex_info.update(self._pex_info_overrides)
		pex_info.pex_path = ':'.join(filter(None, combined_pex_paths))

Improve PEX_PATH handling in pex.py #421

Improve PEX_PATH handling in pex.py #421

Conversation

CMLivingston commented Oct 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CMLivingston Oct 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kwlzn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CMLivingston commented Oct 9, 2017 •

edited

Loading

CMLivingston Oct 11, 2017 •

edited

Loading