Allow multiple interpreters for a pex

[a-recknagel]

Hi,

I think my issue is with documentation, because from reading #654 I feel like my use case seems to be supported, but I couldn't quickly work out how.

---

So, I have an application with a script-entrypoint that I want to bundle with its dependencies to offer my clients as a single "binary" that I know will Just Work on their machine without needing to download anything that could potentially break / is untested.

I have build my application and all its dependencies as wheels, put them in a folder called wheelhouse, and use this command to build the pex file:

pex --interpreter-constraint ">=3.8" -r requirements.txt setuptools my_app -c my_app_entrypoint --no-pypi --repo wheelhouse/ -o my_app.pex

I know that it uses a python version 3.8 for the pex command, because it's the only interpreter on the build system, but the application should run just fine under python3.9 as well (which is what I wanted to express with >=3.8). It works fine with python 3.8, but when I run it with only a python3.9 on my PATH, it fails. How can I correctly tell pex during the build that it's ok to use 3.8 or higher?

[jsirois]

@a-recknagel it appears your PEX includes at least one platform-specific wheel. You can confirm this with pex-tools <your PEX> info -i2. You should see in distributions at least one with platform specific tags (like cp38m). If you want to build a PEX that includes the distributions needed by interpreters you don't have on hand, you need to specify a --platform for each and every one of those. To see how to write a --platform, consult pex --help which will include an example for your current platform. Alternatively, use pex --platform help.

[a-recknagel]

Yeah, that's probably it, I didn't really consider that. Can I create a multi- platform pex by including e.g. cp38 and cp39 wheels of such dependencies in a single one, or should I create multiple pexes instead?

[jsirois]

You can create one PEX with all the wheels. If you specify multiple platforms (--platform X --platform Y ...), Pex does the right thing. You can again inspect this with pex-tools <your PEX> info -i2 after building the PEX.

[a-recknagel]

Sweet, thanks again. I'll post the right command as soon as I worked it out, in case anyone stumbles over this post via google.

[a-recknagel]

I might need some handholding. Is there something like a discord or gitter for pex?

[jsirois]

Pex falls under the pantbuild community. More here: https://www.pantsbuild.org/docs/getting-help

In short though we have a public slack presence at pantsbuild.slack.com that you can join via https://join.slack.com/t/pantsbuild/shared_invite/zt-d0uh0mok-RLvVosDiX6JDpvStH~bFBA. There is no dedicated #pex channel so you should just ask questions on #general. You'll want to have in-hand a list of the --platforms you're targeting, hopefully obtained by running pex --help and recording the platform noted in the --platform help in a venv that uses the interpreter you wish to target on each machine / platform you wish your application to run on.

[a-recknagel]

refined in #1488, fixed by #1489

[a-recknagel]

As promised, the command I ended up using with some explanations as to why certain arguments / options had to be there. Note that the way I ended up building the command depends very heavily on the fact that I provide all 1st and 3rd party packages up-front in a wheelhouse, so there is no ad-hoc building or talking to pypi during the pex-command:

$ pex  # base command
    setuptools my_app ... sqlalchemy        # list of dependencies, -r requirements.txt doesn't work well with 
                                            # multi-platform builds because it contains version-specific info.
                                            # for some reason, you always need to add setuptools as well
    -c my_app                               # name of the script-entrypoint that installing my cli-app provides
    --no-pypi                               # don't download dependencies from the internet, since ... 
    --repo wheelhouse/                      # ... they're all supposed to be in here, multiple versions if
                                            # they are version-specifc (e.g. sqlalchemy)
    --platform linux_x86_64-cp-39-cp39      # listing all platforms I want to support
    --platform linux_x86_64-cp-38-cp38      # see https://www.python.org/dev/peps/pep-0600/ for info
    --interpreter-constraint '>=3.8,<3.10'  # not 100% sure I really need this, but it gives a good error 
                                            # message in case a fitting interpreter can't be located
    --python-shebang "/usr/bin/env python3" # very important, the default shebang doesn't work with venvs
    -o my_app-0.1.0.pex                     # name of the target .pex file

In case it isn't clear how to build a wheelhouse, after generating all requirements for one platform into a requirements.txt, I used pip wheel -w wheelhouse -r requirements.txt to add those dependencies to a wheelhouse folder (the name is just convention, it could have any other).