Add first-class support for multi-interpreter and multi-platform pex construction. #394
Conversation
kwlzn
force-pushed
the
kwlzn/multiplatform_pex
branch
6 times, most recently
from
beb3132
to
85b8a75
Compare
kwlzn
force-pushed
the
kwlzn/multiplatform_pex
branch
from
85b8a75
to
c8b5bdf
Compare
| @@ -290,8 +293,11 @@ def configure_clp_pex_environment(parser): | |||
| group.add_option( | |||
| '--platform', | |||
| dest='platform', | |||
| default=Platform.current(), | |||
| help='The platform for which to build the PEX. Default: %default') | |||
| default=[], | |||
There was a problem hiding this comment.
How about [Platform.current()] here to make the help clearer (%default)? I'd be game for the duplicate [Platform.current()] defaulting code in resolve_multi if extracting a default_platforms function to some reasonable location for both to use seems onerous / more trouble than it's worth.
There was a problem hiding this comment.
ah, how would you feel about just replacing the %default here with a literal description?
the problem with defaulting to [Platform.current()] here is that the action=append mode is additive only, so without implementing a custom callback action for this option you'd end up resolving/building for the local platform every time whether or not you actually intended to.
pex/bin/pex.py
Outdated
| with TRACER.timed('Resolving interpreter', V=2): | ||
| interpreter = interpreter_from_options(options) | ||
| def _lowest_version_interpreter(interpreters): | ||
| """Given an iterable of interpreters, return the one with the lowest version.""" |
There was a problem hiding this comment.
More than an iterable. Just say list?
pex/bin/pex.py
Outdated
| """Given an iterable of interpreters, return the one with the lowest version.""" | ||
| lowest = interpreters[0] | ||
| for i in interpreters[1:]: | ||
| lowest = lowest if lowest < i else i |
There was a problem hiding this comment.
This implies the interpreter objects implement < which suggests sorted(interpreters)[0] (or -1, I'm never sure!). If this thinking is correct, then the comment above is moot.
There was a problem hiding this comment.
I considered that and attempted it, but given that PythonInterpreter only implements __lt__ I was worried that sorted could misbehave - so stuck to use of the bare < operator only here.
I tried to add functools.@total_ordering to PythonInterpreter, which should've made sorted 100% safe, but that failed in the 2.6 shard due to not being present - so stuck with a 2.6-safe approach for the time being.
This PR adds a new CLI option, -p or --preamble-file, that is used to provide a preamble to the pex builder object. We use pex via its API in the internal build tooling at NerdWallet (👋, we're neighbors!). We've been leveraging it for multi-platform support, but now that #394 / 1.2.9 has landed the only thing missing from the CLI that we leverage is the preamble in the builder, which was pretty easy to hook up. Once this PR has landed we can remove all of the API integration and just rely on the pex CLI! The CLI option and the PEXBuilder preamble functionality are covered by tests. Local functional testing: (pex-foo) evanborgstrom@evanborgstrom /tmp/preamble — u:34 j:1 (21:49:54 08.16) #536 ❯❯❯ echo 'print "foo!"' > preamble (pex-foo) evanborgstrom@evanborgstrom /tmp/preamble — u:34 j:1 (21:50:11 08.16) #537 ❯❯❯ pex -p preamble foo! Python 2.7.13 (default, Jun 7 2017, 11:02:53) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> (pex-foo) evanborgstrom@evanborgstrom /tmp/preamble — u:34 j:1 (21:50:18 08.16) #538 ❯❯❯ pex Python 2.7.13 (default, Jun 7 2017, 11:02:53) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>>
This revives and extends the prior attempt in #201 to forklift the multi-interpreter/platform resolve behavior from pants directly into pex to better support general usage via
pex(1)(and transitively, the Heron team's Bazel+pex integration).Fixes #68.