Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pex should support creating a venv directly #1752

Closed
zmanji opened this issue May 9, 2022 · 3 comments · Fixed by #2140
Closed

Pex should support creating a venv directly #1752

zmanji opened this issue May 9, 2022 · 3 comments · Fixed by #2140
Assignees

Comments

@zmanji
Copy link
Collaborator

zmanji commented May 9, 2022

With the lockfile and venv features it's very easy now to create reproducible venvs with pex. It's very handy for setting up development environments without Docker and you can ensure all of the dependencies are wheels and not sdists.

I have been doing this:

  1. Create a lock file with pex3 lock create. This lockfile contains the resolves for all third party dependencies for all platforms I care about. I can use --no-build to ensure all of the dependencies are wheels.
  2. With the above lockfile create a dependencies pex with pex --lock pex.lock --venv -o depencies.pex
  3. Create the venv: PEX_TOOLS=1 ./dependencies.pex venv ./venv
  4. Iterate on my first party code by running PEX_EXTRA_SYS_PATH='./srcs' ./out.pex.venv/pex -m 'mymodule'. In my case srcs contains multiple packages.

I install the virtualenv for two reasons:

  1. Having a virtualenv allows for tools like PyCharm to pick up all of my dependencies.
  2. On my machine, I have observed that executing the pex directly has about 30ms of overhead which is just enough for me to go through with step 3 above.

I think pex should be able to collapse steps 2 and 3 above into a single command invocation. I would envision something like:

pex3 venv create --lock pex.lock -o venv
@jsirois
Copy link
Member

jsirois commented May 9, 2022

Although you can command line golf today:

$ pex3 lock create --no-build cowsay -o lock.json
$ PEX_TOOLS=1 pex --lock lock.json --include-tools -- venv cowsay.venv
$ ./cowsay.venv/bin/cowsay Obscure
  _______
| Obscure |
  =======
       \
        \
          ^__^
          (oo)\_______
          (__)\       )\/\
              ||----w |
              ||     ||

I agree this would be useful. Your suggestion of pex3 venv ... sounds about right as a place for this to live.

@jsirois
Copy link
Member

jsirois commented May 9, 2022

FWIW @zmanji the new --sh-boot combined with --venv should drop the overhead of warm invocations over native venv to ~1ms. So FYI for those cases where you don't need to know the venv path to tell an IDE.

jsirois added a commit to jsirois/pex that referenced this issue Apr 24, 2023
This allows creating venvs (or updating existing ones) directly from
locks or subsets of locks.

Fixes pex-tool#1752
@jsirois jsirois self-assigned this Apr 24, 2023
@jsirois
Copy link
Member

jsirois commented Apr 24, 2023

I ended up providing pex3 lock {export,subset,venv} ... instead. The export sub-command already existed but now supports passing -r requirements files to use to subset the lock. The subset and venv commands are new and both support optional sub-setting. The subset sub-command creates a new lockfile with a transitive closure subset of the original lock file. The venv sub-command is just like the PEX_TOOLS venv tool, except that it is faster. I'm using the latter with nox which has both sped up session set up enough that running my full suite is now ~40% faster than using nox's session.install(...) and made all tooling and packaging reproducible. PR(s) coming shortly.

jsirois added a commit to jsirois/pex that referenced this issue Apr 28, 2023
This command allows inspecting venvs created by Pex as well as those
created by other tools.

Work towards pex-tool#1752 and pex-tool#2110
jsirois added a commit that referenced this issue Apr 28, 2023
This command allows inspecting venvs created by Pex as well as those
created by other tools.

Work towards #1752 and #2110
jsirois added a commit to jsirois/pex that referenced this issue Apr 28, 2023
This code as well as selection of a unique target to resolve for will
be re-used by the forthcoming `pex3 venv create` command.

Along the way, clean up confusion noted in pex-tool#2135 in the introduction
of the `pex3 venv` subcommand surrounding `pyvenv.cfg` files generated
by `Virtualenv`.

Work towards pex-tool#1752 and pex-tool#2110.
jsirois added a commit that referenced this issue Apr 28, 2023
This code as well as selection of a unique target to resolve for will be
re-used by the forthcoming `pex3 venv create` command.

Along the way, clean up confusion noted in #2135 in the introduction of
the `pex3 venv` sub-command surrounding `pyvenv.cfg` files generated by
`Virtualenv`.

Work towards #1752 and #2110.
jsirois added a commit to jsirois/pex that referenced this issue Apr 30, 2023
This new subcommand can create either a venv or just populate a flat
`sys.path` directory entry (ala `pip install --target`) given a set of
requirements to resolve, potentially from a lock or an existing PEX
file, but otherwise from indexes and find links repos. Unlike the
sibling `venv` `pex-tool` subcommand, the target can be selected and,
in the flat `sys.path` directory entry case, it can be a foreign
platform.

Fixes pex-tool#1752
Fixes pex-tool#2110
jsirois added a commit to jsirois/pex that referenced this issue May 1, 2023
This new subcommand can create either a venv or just populate a flat
`sys.path` directory entry (ala `pip install --target`) given a set of
requirements to resolve, potentially from a lock or an existing PEX
file, but otherwise from indexes and find links repos. Unlike the
sibling `venv` `pex-tool` subcommand, the target can be selected and,
in the flat `sys.path` directory entry case, it can be a foreign
platform.

Fixes pex-tool#1752
Fixes pex-tool#2110
jsirois added a commit that referenced this issue May 1, 2023
This new sub-command can create either a venv or just populate a flat
`sys.path` directory entry (ala `pip install --target`) given a set of
requirements to resolve, potentially from a lock or an existing PEX
file, but otherwise from indexes and find links repos. Unlike the
sibling `venv` `pex-tool` sub-command, the target can be selected and, 
in the flat `sys.path` directory entry case, it can be a foreign 
platform.

Fixes #1752
Fixes #2110
Fixes #2111
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants