distutils.cfg Can Break venv

[nicksloan]

BPO	17732
Nosy	@birkenfeld, @vsajip, @larryhastings, @tarekziade, @carljm, @merwok
Files	distutilsvenv.patch: distutils/dist.py patch for venv distutilsvenv.patch: Updated patch. distutilsvenv.patch distutilsvenv.patch distutilsvenv.patch distutilsvenv.patch distutilsvenv.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/merwok'
closed_at = <Date 2013-05-12.11:23:17.029>
created_at = <Date 2013-04-14.18:10:32.241>
labels = ['type-bug', 'library', 'release-blocker']
title = 'distutils.cfg Can Break venv'
updated_at = <Date 2013-05-16.17:57:20.161>
user = 'https://bugs.python.org/nicksloan'

bugs.python.org fields:

activity = <Date 2013-05-16.17:57:20.161>
actor = 'georg.brandl'
assignee = 'eric.araujo'
closed = True
closed_date = <Date 2013-05-12.11:23:17.029>
closer = 'georg.brandl'
components = ['Distutils']
creation = <Date 2013-04-14.18:10:32.241>
creator = 'nicksloan'
dependencies = []
files = ['29870', '29873', '29888', '29889', '29939', '29940', '29976']
hgrepos = []
issue_num = 17732
keywords = ['patch']
message_count = 24.0
messages = ['186942', '186943', '186945', '186973', '186978', '187004', '187010', '187012', '187018', '187021', '187111', '187112', '187113', '187296', '187303', '187363', '187376', '187386', '187563', '188446', '188546', '189011', '189387', '189393']
nosy_count = 9.0
nosy_names = ['georg.brandl', 'vinay.sajip', 'larry', 'tarek', 'carljm', 'eric.araujo', 'python-dev', 'nicksloan', 'eruart']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17732'
versions = ['Python 3.3', 'Python 3.4']

[nicksloan]

When distutils.cfg defines an install-lib, it will be used within virtual environments created using venv as well, which makes it impossible to install things with setup.py or similar.

Steps to reproduce:

Create a distutils.cfg file and set a value for install-lib.

Create a virtual environment: $pyvenv myvenv

Activate that environment: $source myvenv/bin/activate

Attempt to install something: $python distribute_setup.py

It will attempt to install in the lib-install directory, and should fail with an error that the directory is not on the python path. This issue affects python3 from the mac homebrew project, which bundles a distutil.cfg, thus breaking pyvenv by default.

[merwok]

There is a --no-user-cfg option to disable reading ~/.pydistutils.cfg, but I don’t recall an option to ignore the global config. How does virtualenv avoid this problem, if it does?

[nicksloan]

It looks like virtualenv includes an empty distutils.cfg at myvirtualenv/lib/python3.3/distutils/distutils.cfg. Most of myvirtualenv/lib/python3.3 is just symlinked to the base python version, but myvirtualenv/lib/python3.3/distutils/ is not, and I bet this is exactly why.

[vsajip]

This looks to me as if it will need a patch in distutils. Unlike virtualenv, which contains a patched copy of distutils (and hence allows having a .cfg adjacent to it), pyvenv does not create patched modules in the venv. It does not make sense to change this behaviour.

The correct solution would appear to be for distutils to ignore certain configuration options (install-lib, but also equivalent options for headers and scripts) when running in a venv.

[nicksloan]

That's along the lines of what I've been thinking as I dig into this. I'd love to take a stab at a patch for this if no one else has done so already.

[nicksloan]

Here is a patch that seems to fix the problem. It simply short-circuits distutils options that change directories. This is my first python patch ever, so I'm eager for comments. Is this the right approach?

[nicksloan]

On second thought, there is probably no good reason to ignore the build-* settings. Here is an updated patch.

[vsajip]

On second thought, there is probably no good reason to ignore the build-* settings.

I was just about to mention this. I'm not an expert on distutils internals and whether this is the best way to accomplish what's needed, but the approach seems reasonable to me. Why the empty list and the extend, though? Why not something like the following?

if sys.prefix == sys.base_prefix:
    # not in venv
    ignore_options = []
else:
    # in venv
    ignore_options = ['install-base', ...]

My other comments on the patch are generic, i.e. you need to look at the tests and docs, too.

Anyway, thanks for the effort you've spent to come up with a patch. Can I suggest that you sign a PSF contributor form to license your work to the PSF?

http://www.python.org/psf/contrib/contrib-form/

[nicksloan]

My thought was that perhaps there will be other circumstances where we may want to ignore options in the future. The idea was that by providing an ignore_options list that can be extended, multiple conditions with different sets of options can be stacked together easily. I was thinking a set might actually be an even better choice than a list.

Would love feedback on this approach. Am I over-designing for an unlikely case? I'm happy to use Vinay's suggestion if we don't think it is worth it to consider possible future conflicts with distutils.cfg options. If there is any reason a set would be worse to use, let me know, otherwise my next patch will use a set rather than a list.

Another question: should I be checking which config file these come from, or ignoring all occurrences of these options? Seems like setup.cfg at the very least shouldn't have any restrictions. Possibly ~/.pydistutils.cfg too.

I'll review tests to see if this necessitates any changes, add a new test that covers this, and I'll make a note in the docs.

[merwok]

I would ignore options from all config files, and do the simplest thing (i.e. not add the ignore_options attribute).

[nicksloan]

Here is an updated patch with documentation changes and a new test. 5 tests in distutils have errors. I have left those alone for now.

[nicksloan]

That is, errors that pre-existed my patch.

[nicksloan]

Whoops. One of the options I had in my list doesn't actually exist. Here is yet another update.

[nicksloan]

Any feedback on this latest patch?

[merwok]

Thanks for the patch, I left comments on rietveld.

[nicksloan]

Responded to comments with an updated patch. Thanks for all the feedback, and sorry for the silly mistakes. Should have read up more thoroughly on the docs style guide and the terminology. Hopefully the latest patch is ready to go (or at least, nearly so).

[nicksloan]

Here is another update. It has come to my attention that I missed some options:

prefix, exec-prefix, home, user and root

These have been added, and the docs and test have been updated to reflect the change.

[merwok]

Looks good, thanks! I assume you also tested it manually?

I’ll take care of this for the next release.

[nicksloan]

Éric,

Fixed the mention of packages, and made a frozenset of ignore_options. Think this thing is ready to go. I have tested it, and it does the trick. Thanks for helping me get this patch merged in.

[nicksloan]

Just checking to see if anything else is needed from me on this.

[merwok]

No, patch is good to go, I just need to find some time to commit it. If another core dev wants to do it sooner, please feel free to do so.

[python-dev]

New changeset 6c5a3d194a10 by Georg Brandl in branch '3.3':
Closes issue bpo-17732: ignore install-directory specific options in
http://hg.python.org/cpython/rev/6c5a3d194a10

[python-dev]

New changeset d62f71bd2192 by Brian Curtin in branch '3.3':
Add Nick Sloan for his contribution to bpo-17732
http://hg.python.org/cpython/rev/d62f71bd2192

[birkenfeld]

Thanks for the attribution, that was definitely an oversight on my part.