Provide a get-pip.py-like script for virtualenv

[asottile]

I often find myself on a machine with near-zero permissions (or a chroot / docker container) and have used this script: https://github.com/asottile/asottile.github.io#bootstrap-a-virtualenv-from-nothing

Would love to get this or something like this hosted similar to get-pip.py

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.

[asottile]

I still think this is a good idea

[gaborbernat]

@asottile make a PR with tests, add it to the documentation and we're ready then 👍

[asottile]

sounds good! as for design there's kind of two options -- which would you prefer:

1. like get-pip.py: bundle the actual bits in the file itself
2. like my script I linked above, reach out to pypi to download the tgz / wheel

(1) seems like a better idea but would like to get your opinion before working on it

[gaborbernat]

I would prefer the second, cleaner. Note we might need to call a wheel build though, going ahead we no longer support sdist evaluation (as I'd like the horrible one file we have now split up and concatenated during build rather than on the repo, this would drop need of doing current embedded hack). That being said would it be possible to use pypi wheel as download as opposed to a GitHub file?

[asottile]

option 1 is nice though if you don't have a network connection (think: vendor file in repo, able to create venvs without dependencies) -- doing the wheel would be fine as well (I was just using the sdist in my script for no good reason at the time)

[gaborbernat]

I would still go down the path of wheel extract, python can natively unzip it. I mean the user will have to get the repo in some way or another, so some network must exist, not?

[asottile]

Unless you're on macos :( it's crippled ssl implementation can't talk to pypi by default with just stdlib

[gaborbernat]

isn't github also https?

[asottile]

Yes but (for now) doesn't require tls 1.2 or whatever

[asottile]

And curl can take advantage of the other ssl implementation available on macos (pip does this) but urllib hasn't figured that out yet

[gaborbernat]

Oh I see, in that case we just need to use either pip or curl, not?

[asottile]

Then it fails the "bootstrap from just python" aspect :S

[gaborbernat]

Somehow you need to get the bootstrapper on the machine though, not?

[asottile]

yeah, but once you have the bootstrapper it should be able to do everything itself -- even my script above fails that category until you've got a decently-compiled python on macos (brew for instance)

this is why I'm leaning towards option 1

[gaborbernat]

Isn't the wheel (perhaps wrapped beside a simple python script) able to do everything?

[asottile]

yes, that's what I'm proposing :)

[gaborbernat]

I'm on board with that. It could be another artefact we build upon release? Should we publish it on PyPi, or Github?

[asottile]

I was hoping it could end up on https://bootstrap.pypa.io/get-virtualenv.py (similar to https://bootstrap.pypa.io/get-pip.py)

[gaborbernat]

Let's have it ready then, and we'll try to get it up there. That being said from what we talked wouldn't it be https://bootstrap.pypa.io/get-virtualenv.zip?

[asottile]

could either be .py or .zip -- though I haven't done anything with PEP 441 yet (and from cursory glance it doesn't support py2). I was more thinking the same approach that get-pip.py uses -- it embeds the .whl in a string literal

[ncoghlan]

The zipapp format has been supported since Python 2.6 - the only new parts in PEP 441 were the zipapp helper library for creating them, and the pyz file association on Windows.

[asottile]

oh perfect, that makes this even easier I think

[ncoghlan]

Yeah, I think Py2.4 was still in broad use back when get-pip.py was designed, so you have nicer options available to you now :)

[pradyunsg]

If someone does the work for packaging virtualenv in a zipapp for installation, it might be useful to consider adopting the same approach for pip too. Though, I'm not sure how the transition would be done.

"Modernize get-pip.py" has been in my mental TODO list since pip 10.0, but I never got to it because of the effort-reward ratio (low reward, high effort "fun" task).

[pfmoore]

For information, it's possible to bundle virtualenv up using the shiv tool. I use this for a number of applications (virtualenv, tox, pytest, pipenv, ...) and it works really well.

I've not tried it for pip, but it may well work there - either for a standalone pip, or as a get-pip equivalent...

[asottile]

got it working, would like some feedback before I write up news entries / decide further on the approach: pypa/virtualenv#1287

[gaborbernat]

We now have https://github.com/pypa/get-virtualenv, we just need to follow the logic from https://github.com/pypa/get-pip and deploy the zipapp generated from here 👍

[asottile]

I want to pick this up again and make it happen -- what needs doing here 😁

[asottile]

I think for virtualenv we don't need any of the complexity of get-pip -- make_zipapp.py builds virtualenv.pyz for us which I think (?) is all that's necessary

[pfmoore]

I guess there are 2 different possibilities:

1. You want a "single file executable" version of virtualenv, that you can just run with any Python interpreter that you have to hand, without "installing" it in any way. virtualenv.py is almost that, but it needs the virtualenv_support directory to bootstrap pip, setuptools and wheel. Bundling virtualenv.py and the support directory in a zipapp does the job here, though.
2. You want an installer for virtualenv, that can be used without having pip installed. That's basically what get-pip.py is, but the need for this is more important for pip, where it's a bootstrapping issue.

I assume what's wanted here is (1), and a pyz is a fine solution for that. Most of the complexities of get-pip.py are to solve bootstrapping issues that no-one but pip has.

[asottile]

yeah, I already did the work to make the zipapp for this purpose (1) -- I believe all that's needed is a place to stick it and docs that point at it (and a procedure to update it)

[pradyunsg]

@asottile We're on the issue tracker of exactly that repository :P

[pradyunsg]

Oh, nvm me. I need sleep. 😴

[anthrotype]

I was hoping it could end up on https://bootstrap.pypa.io/get-virtualenv.py (similar to https://bootstrap.pypa.io/get-pip.py)

I was hoping too 👍
Is this still planned?

[gaborbernat]

I believe so 🤔 though maybe a zipapp is enough here, what's your use case?

[asottile]

yeah the zipapp should be enough, just need to get it hosted

[gaborbernat]

So I'll try to set this up with virtualenv-20.0.0b2.pyz.

[gaborbernat]

So here's the proposal. Once this ticket is delivered we'll enable the following functionality:

# download a self-packaged virtualenv
curl http://bootstrap.pypa.io/get-virtualenv.pyz
# point a python to it, and you're ready to create virtual environments
python get-virtualenv.pyz --help

Proposed behaviour:

• part of every virtualenv release will be to publish the generated zipapp to this repository (~6MB file). The latest zipapp will always live at the root of this repository.
• furthermore, we'll have the latest supported zipapp per python micro version via a directory structure; e.g: 3.8/virtualenv.pyz will be a virtual zipapp that's supported under python3.8. Initially, these will just symlink to the root level (as initially, we support all those Pythons); however as we start dropping support for given versions these links might contain the last supported version.

get-virtualenv/
├── virtualenv.pyz
├── 2.7
│   └── virtualenv.pyz
├── 3.4
│   └── virtualenv.pyz
├── 3.5
│   └── virtualenv.pyz
├── 3.6
│   └── virtualenv.pyz
├── 3.7
│   └── virtualenv.pyz
├── 3.8
│   └── virtualenv.pyz
└── 3.9
    └── virtualenv.pyz

• http://bootstrap.pypa.io/get-virtualenv.pyz will point to the virtualenv.pyz at the root level,
• http://bootstrap.pypa.io/3.8/virtualenv.pyz will point to the 3.8/virtualenv.pyz file within the repository.

@ewdurbin can help set up this link from http://bootstrap.pypa.io to this repositories contents. Is everyone happy with this suggestion?

[asottile]

sounds good, looks a lot like how get-pip is set up

[ewdurbin]

Currently this will end up being a bit of a headache to maintain.

bootstrap.pypa.io currently pulls in three repositories

• https://github.com/pypa/get-pip/tree/master => /srv/bootstrap/pip
• https://github.com/pypa/setuptools/tree/bootstrap => /srv/bootstrap/setuptools
• https://github.com/buildout/buildout/tree/bootstrap-release => /srv/bootstrap/buildout

and merges them via symlinks:

ernestd@web0:/srv/bootstrap/www$ tree -l
.
├── 2.6
│   └── get-pip.py -> /srv/bootstrap/pip/2.6/get-pip.py
├── 3.2
│   └── get-pip.py -> /srv/bootstrap/pip/3.2/get-pip.py
├── 3.3
│   └── get-pip.py -> /srv/bootstrap/pip/3.3/get-pip.py
├── 3.4
│   └── get-pip.py -> /srv/bootstrap/pip/3.4/get-pip.py
├── bootstrap-buildout.py -> /srv/bootstrap/buildout/bootstrap/bootstrap.py
├── ez_setup.py -> /srv/bootstrap/setuptools/ez_setup.py
└── get-pip.py -> /srv/bootstrap/pip/get-pip.py

4 directories, 7 files

You can see the way it is currently configured/managed here.

Adding virtualenv to the mix in this way will require all kinds of additional symlinks and purges and kind of makes me think it's time to rethink the way that bootstrap.pypa.io is constructed a bit.

It seems setuptools and buildout just have a single file, so I guess they can remain the same at the moment, but for get-pip and get-virtualenv I think it would make sense for us to restructure a bit by adding a publish directory to the top level of each repository that includes files that it would like to serve.

For get-pip that directory would basically be

publish/
├── 2.6
│   └── get-pip.py
├── 3.2
│   └── get-pip.py
├── 3.3
│   └── get-pip.py
├── 3.4
│   └── get-pip.py
└── get-pip.py

and for get-virtualenv, based on @gaborbernat's sketch above:

publish/
├── virtualenv.pyz
├── 2.7
│   └── virtualenv.pyz
├── 3.4
│   └── virtualenv.pyz
├── 3.5
│   └── virtualenv.pyz
├── 3.6
│   └── virtualenv.pyz
├── 3.7
│   └── virtualenv.pyz
├── 3.8
│   └── virtualenv.pyz
└── 3.9
    └── virtualenv.pyz

Then, give each a link with their name in the top level of bootstrap.pypa.io:

.
├── pip -> /srv/bootstrap/pip/publish
│   ├── 2.6
│   │   └── get-pip.py
│   ├── 3.2
│   │   └── get-pip.py
│   ├── 3.3
│   │   └── get-pip.py
│   ├── 3.4
│   │   └── get-pip.py
│   └── get-pip.py
└── virtualenv -> /srv/bootstrap/virtualenv/publish
    ├── 2.7
    │   └── virtualenv.pyz
    ├── 3.4
    │   └── virtualenv.pyz
    ├── 3.5
    │   └── virtualenv.pyz
    ├── 3.6
    │   └── virtualenv.pyz
    ├── 3.7
    │   └── virtualenv.pyz
    ├── 3.8
    │   └── virtualenv.pyz
    ├── 3.9
    │   └── virtualenv.pyz
    └── virtualenv.pyz

In an ideal world, that'd be that! If you want to boostrap pip, curl https://boostrap.pypa.io/pip/get-pip.py

But for backwards compatibility, we'd need to reconstruct the existing links. Some automation could scan the publish directories to create the top level version directories of existing projects, but if we want to make adding additional projects to boostrap.pypa.io a thing, it might make sense to move away from everything sharing the top level.

Would urls like https://bootstrap.pypa.io/virtualenv/virtualenv.pyz or https://boostrap.pypa.io/virtualenv/3.8/virtualenv.pyz be a problem?

[ewdurbin]

Alternatively, I suppose we could leave the "latest and greatest" in the top level https://bootstrap.pypa.io/virtualenv.pyz and for accessing specific versions https://boostrap.pypa.io/virtualenv/3.8/virtualenv.pyz

[ewdurbin]

@dstufft @pradyunsg @pfmoore it looks like y'all have been active in preparing/releasing things to bootstrap.pypa.io for get-pip and setuptools, thoughts on this?

[gaborbernat]

I'm personally happy with both proposals. What I'd like to achieve here is to have a link at root level-ish, that points to the latest. Then have one more in a subdirectory (path in case of URL), that would link to the latest that supports that major.minor combo. Other than that whatever @dstufft, @pradyunsg or @pfmoore is fine by me 👍

[pradyunsg]

Ah, I thought I'd already responded (twice!)

Overall, I like the idea of making a directory in the repository be the public-facing directory, instead of depending on finer details of the repository's layout. The main change I suggest is that we call the directory public and not publish (mostly because that's common on web-stuff I've worked on, and the icon theme on my editor has a nice icon for that).

---

I think we should probably discuss the get-pip specific things (like maintaining existing routes) on the get-pip issue tracker: I've filed pypa/get-pip#61 for that.

[pradyunsg]

Noting that we'd probably want to add digests and/or signatures for the files as well. This reorganization has the benefit of making that easier to deploy/work on. (see pypa/get-pip#47 and pypa/get-pip#41)

[gaborbernat]

I've added to this repository the files as suggested, https://github.com/pypa/get-virtualenv/tree/master, version 20.0.0b2, see #2.

[ewdurbin]

@gaborbernat assuming that some tooling will eventually be added to the repository, I think the structure needs to be changed a bit to have a public directory (as @pradyunsg has recommended for the name). That directory should have what we actually want to serve on bootstrap.pypa.io.

[gaborbernat]

@ewdurbin just to confirm, should we move everything as it is within a public folder at the root of the repository? 👍

[ewdurbin]

I believe so, with the exception of the README

[gaborbernat]

Done via #3

[ewdurbin]

OK, this is being served on bootstrap.pypa.io as of python/psf-salt@ab57d3e

@gaborbernat it looks like the symlinks in git weren't correct though, I resolved that in #4

[gaborbernat]

This is now live, thanks for all help, especially @ewdurbin !

• https://bootstrap.pypa.io/virtualenv.pyz
• https://bootstrap.pypa.io/3.8/virtualenv.pyz

[xavfernandez]

Nice 👍
The version specific link seems to be https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz