added: --install-option on requirements file #271

[conrado]

Hi,

I'm not really sure this is the best way about it... but it works.

Bye.

[hltbra]

Someone asked about it in StackOverflow: http://stackoverflow.com/questions/10278708/pip-pass-custom-options-to-installer-with-requirements-txt

After merging it would be nice to write an answer there.

[bernardobarreto]

@hltbra, I already commented the question in stackoverflow saying about this pull request.

[conrado]

If anyone could review this bad comment line and tell me if there's a better way to go about this I can go and fix it. argparse seemed too complicated.

Example requirements.txt and caveats

pyzmq>=2.2.0 --install-options="--zmq=/usr/local/lib"

the string --install-options= is searched for on the line, and chomped. Doesn't do any error checking, we just grab whatever is inside the quotes. There's no escaping double-quotes, or spaces... which is the next split() in L#90 so you can pass multiple parameters.

So, you could have:
--install-options="--libjpeg=/usr/local/lib --libpng=/usr/local/lib"
but
--install-options="--libjpeg=/home/conrado/library\ builds/"
will fail in a very bad way. So, is this acceptable?

[pnasrat]

Thanks for the pull request.

I'd say we definitely should have tests around this, and this should be part of your pull request. If you need a hand with that just ask and I'll try help.

So shell parsing and escaping is tricky. For --install-options="--libjpeg=/home/conrado/library\ builds/" does --install-options="--libjpeg=/home/conrado/library\\ builds/" work?

Also have you looked at shlex in the stdlib as this may help with splitting

Also we'd definitely want to support this both in requirements.txt and on the command line, so this is probably the wrong place for the support to be added.

[qwcode]

as for parsing the options from the req, pip is already using optparse internally, and it's supported across all the python versions pip supports

here's a way to realiably parse the req using that.

import optparse
p = optparse.OptionParser()
p.add_option("--install-options")
options, args = p.parse_args(args=['pyzmq>=2.2.0', '--install-options="--zmq=/usr/local/lib"'])

the result:
args = ['pyzmq>=2.2.0']
options.install_options = '"--zmq=/usr/local/lib"'

[pnasrat]

This is a thought I'm having that I want to preserve in the issue - thinking of other approaches to how to do this, eg we could on the command line using the standard -- to tell pip to stop parsing options and then have a second parse phase to pullout the options for setup.py.

A quick test shows this seems to work on 2.4 and 2.7. I'm not totally sold on that though so

import optparse
parser = optparse.OptionParser()
parser.add_option('--noarg', action="store_true", default=False)
parser.add_option('--witharg', action="store", dest="witharg")
parser.add_option('--witharg2', action="store", dest="witharg2", type="int")
print parser.parse_args([ '--noarg', '--witharg', 'val', '--witharg2=3' ])
(<Values at 0x106018128: {'noarg': True, 'witharg': 'val', 'witharg2': 3}>, [])
print parser.parse_args([ '--noarg', '--witharg', 'val', '--', '--witharg2=3' ])
(<Values at 0x106018248: {'noarg': True, 'witharg': 'val', 'witharg2': None}>, ['--witharg2=3'])

[conrado]

The issue here is that the argument parsing from shell and from the requirements.txt file are "decoupled"(?) So if we change the shell argument options we need to further change it here. I think @pnasrat is addressing this issue with the -- argument end marker suggestion, however that won't really help I think.

Question on functionality, should we have all pip shell options available to requirements.txt? I don't know this, but some options might not be relevant. If we do want to couple the argument parsing on requirements.txt and the shell arguments, we might have to refactor some code and come up with a more elegant pattern than the current solution.

[pnasrat]

So from my perspective requirements.txt should be able to express everything that is required to install a package. If we need to pass specific options through to setup.py.

@ianb has added another way to do this in #519 although I think you're right we should take a step back and think about what we need to support to fully work. Other cases such as @ niedbalski adding subdirectory support but via the URL #526 point to the need to handle the non trivial cases where just running setup.py won't work.

@jezdez @brosner and @carljm any thoughts?

[qwcode]

yes, reusing the IntallCommand parser somehow and applying it to spec lines in requirements.txt (and overriding the CLI options) would be nice.

[conrado]

I have looked into getting more options supported, but it really does need some work on InstallCommand, RequirementSet, possibly parse_requirements.

The way it is now it should work fine as shlex is used to parse the arguments.

[carljm]

Hey all - I don't have strong opinions here. Ideally we'd clean up requirements file parsing in general to not be so janky, and be able to offer a general "any option you can use on the command line you can use in a requirements file" guarantee, but that's going to be a sizable chunk of work and I'm not sure it's the highest priority. In practical terms I'm not too concerned about duplicating option names; we shouldn't really be changing them anyway for back-compat reasons.

In this pull request, it seems odd to me to reuse the InstallCommand option parser when really the only thing we are supporting is install_option. I think I'd prefer to go with @qwcode 's suggestion and just use optparse directly to look for the only option we currently support here.

[qwcode]

the "reusing the IntallCommand parser" comment was about the possibility of supporting all or most of pip.commands.install options in req files, but not meant to block getting in --install-option support.

[carljm]

@qwcode I know, but the current state of the pull request does already reuse the InstallCommand parser, even though none of the options are respected apart from --install-option; that's what I was referring to.

[qwcode]

@carljm , ok, I follow now. didn't really analyze how different this new commit was.
I can see the oddness, but otoh, seems harmless and likable.
I can see maybe wanting to add line by line support for -U and --no-deps

more concerned about having tests for it.

[qwcode]

not following why this is just in the else block? and also not supported for editables.
e.g. we wouldn't be supporting install options for any of these forms

http://stuff.com/my_stash/package.tar.gz
git+http://github.com/frank/frodo@tag#egg=frodo
-e git+http://github.com/frank/frodo@tag#egg=frodo

[conrado]

I had a concern when it came to sifting through the RequirementSet and overriding options passed through the command line. What has authority and how should it behave?

In fact, I think I need to look more at the current -r options.

[qwcode]

closing this due to #790. will be reviewing that in a few days.