You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As originally reported in pantsbuild/pants#11211, if the timing is right and the set of available interpreters that conform to Pex's interpreter constraints is right it appears that the interpreter identification / selection that runs in parallel in the PEX bootstrap phase can lead to Pex PEX failing to run.
This causes pex to be re-imported from the .bootstrap/ instead of from the pex distribution for the Pex PEX, which is disastrous since the .bootstrap/ only contains enough of pex to bootstrap a PEX file and not the whole CLI. The result is something like:
Traceback (most recent call last):
File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 446, in execute
exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 378, in _wrap_coverage
return runner(*args)
File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 409, in _wrap_profiling
return runner(*args)
File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 508, in _execute
return self.execute_entry(self._pex_info.entry_point)
File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 610, in execute_entry
return runner(entry_point)
File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 625, in execute_pkg_resources
runner = entry.resolve()
File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2481, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'pex.bin'
Note that in the chain of events described above, the critical event - the re-import of pexfrom a background thread - is neither observed when instrumenting imports with a sys.path_importer_hook (only a single import of encodings.ascii by encodings via the __import__ function is observed) nor does it appear this should ever happen by inspection. In other words it appears all pex imports happen before the thread is ever spawned.
The text was updated successfully, but these errors were encountered:
As originally reported in pantsbuild/pants#11211, if the timing is right and the set of available interpreters that conform to Pex's interpreter constraints is right it appears that the interpreter identification / selection that runs in parallel in the PEX bootstrap phase can lead to Pex PEX failing to run.
Interpreter selection is started here in
maybe_reexec_pex
:https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/pex_bootstrapper.py#L471-L474
If the current interpreter:
Then this code will short circuit some time after the 1st interpreter is yielded (i.e.: on the second interpreter or greater):
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/pex_bootstrapper.py#L158-L175
Since the short circuit happens on the second or later interpreter, this execute_parallel code is engaged by the lazy iterator and it spawns a single thread to manage concurrent shell-outs:
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/pex_bootstrapper.py#L103-L108
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/interpreter.py#L776-L796
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/jobs.py#L384-L404
That iterator neither auto-cancels nor is cancelled by this Pex bootstrap code. The upshot, is when the code continues on from
maybe_reexec_pex
toPEX.execute()
there can still be background work going on identifying candidate interpreters that will never be picked:https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/pex_bootstrapper.py#L471-L474
This is fine in and of itself, but it appears it can cause an import of
pex
to interleave between the unimports here and thesys.path
re-arrangement below:https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/bootstrap.py#L36-L52
This causes
pex
to be re-imported from the.bootstrap/
instead of from thepex
distribution for the Pex PEX, which is disastrous since the.bootstrap/
only contains enough ofpex
to bootstrap a PEX file and not the whole CLI. The result is something like:Note that in the chain of events described above, the critical event - the re-import of
pex
from a background thread - is neither observed when instrumenting imports with asys.path_importer_hook
(only a single import ofencodings.ascii
byencodings
via the__import__
function is observed) nor does it appear this should ever happen by inspection. In other words it appears all pex imports happen before the thread is ever spawned.The text was updated successfully, but these errors were encountered: