Decide how to bottle Python stuff

[MikeMcQuaid]

I had a chat with @mistydemeo yesterday about bottling Python stuff and I'd love to have some other input.

Basically, as-is depends_on :python now will use the first Python in your PATH. If this is from Homebrew or the system Python there's slightly different behaviour. It's basically now used to set PYTHONPATH correctly and depend on Python if one is needed. Stuff like scons that doesn't install any bindings should not have a depends_on :python and will, as a result, always use a system Python. This case can be trivially bottled.

Where it gets complicated is when we want to start bottling either stuff that installs only bindings (e.g. pyqt) or stuff where the bindings are currently (and, sensibly should be e.g. libxml2) installed by default.

As far as I see it there are several options:

1. depends_on :python stuff that's bottled will always use the system Python (the current behaviour). If people want to build against their own Python we tell them to use --build-from-source.
2. Same as 1. except if the system Python is not first in their PATH we automatically build from source.
3. depends_on :python stuff that's bottled will always use the Homebrew Python and will be built against that. If people want to build against their own or the system Python we tell them to use --build-from-source.
4. Same as 3. except if the Homebrew Python is not first in their PATH we automatically build from source.
5. We never build Python bindings by default; they are all made optional.

Perhaps related to the above: we currently assume the system Python is 2.7. If it's found to not be (either someone has fucked around with it or it's an older version of OSX) then we use the Homebrew Python instead of the system one.

My personal preference here is either 1. or 2. I believe @mistydemeo's was either 3. or 4. I'm somewhat opposed to them for the reason that I'd rather not add a Homebrew Python dependency to e.g. libxml2 just because we want to build the bindings.

If there's any other options here I'm missing, please let me know. @adamv @asparagui @jacknagel @mistydemeo @mxcl @manphiz @samueljohn @chrmoritz @jenshnielsen @daviewales I'd love your thoughts here.

Note: I now hate Python.

[samueljohn]

I am in favour of (2.) and no, I don't see any smart solution.

[MikeMcQuaid]

Another question somewhat related to this: is it safe to just delete all .pyc files before bottling? In the case of e.g. scons they contain the full path to the original source file and are the only thing stopping it from being a cellar :any bottle.

[samueljohn]

Yes, the .pyc are just binary representations of the parsed .py files and are not needed. They are created on-demand (first run). There is even an option to disable the .pyc file generation (by setting some env var).

[MikeMcQuaid]

Thanks @samueljohn. For future reference: you can use python -B or set PYTHONDONTWRITEBYTECODE.

[NikolausDemmel]

As I guess botteling both versions at the same time (i.e. creating two alternative bottles) is not an option, I would be in favour of (2).

Moving the python bindings to seperate formulae probably doesn't make sense/isn't feasable/isn't desired.

With this it should be clarified on the wiki pages relating to python, that as a default for users who are not sure what they need, sticking with system python is recommended. When I first installed homebrew I got the impression that brewed python would be the recommended route (even though it might not say that explicitely anywhere).

[jenshnielsen]

I think either 2 or 4 are fine. The important thing is that homebrew doesn't silently install binding build against a different version of python that the one which is first on the path. If 1 or 3 explicitly fails when the bottled version is different from the one first on path this is also OK (with a helpful error message that is)

Unless all packages are build against both versions it is unavoidable that some users will have to build from source and unless we have any strong indication that one python version is more commonly used than the other I don't see any strong arguments for either 2 or 4.

A note which is perhaps relevant: There are packages such as Pymol in homebrew-python which cannot be build against system python because it needs to be build with brewed-tk.

[ellisonbg]

Pinging @minrk for his thoughts on this.

[wjwwood]

I would prefer 2 because it is more inline with Homebrew's policy of not replacing things Apple provides unless completely necessary. I would also be interested in whether or not multiple bottles (one for each Python) could be possible.

[chdiza]

The choice here, IIUC, is between who is going to have to forgo bottles and build some (sometimes) gigantic packages from source---System Python folks, or Brewed Python folks, or in the case of (5), everyone.

Because bottle-pouring is so convenient in the case of those gigantic things, there is pressure on the user to choose the bottle route. Under (3)/(4), the pressure is towards using a brewed python; it therefore seems to me that (3)/(4) will result in a larger ticket burden (or at least, passes up a chance at a smaller ticket burden), which is one reason to prefer (1)/(2). (Though certainly not a by-itself-decisive reason).

Why do I think that will result in a larger ticket burden? Because in monitoring the issues page for over a year now, Python is the single biggest cause of headache in brew land. If you look at the "labels" breakdown, it's always by far the most populated tag of any of the package-specific tags (i.e., not generic ones like "new formula" or "usability"). The vast majority of these seem to be about brewed Python itself or about other packages getting confused wrt brewed vs. system Python.

This is despite smart people obviously working very, very hard---heroically, in fact---to make Python play nicely with Homebrew. But Python is like the Hydra to Homebrew's Heracles(es). It seems more sensible for the default to be the "safer" thing.

[MikeMcQuaid]

Thanks for the input everyone.

Glad to see 2. looking quite popular. It's my preferred option too, I think.

@chdiza While I do agree with you to present the opposite viewpoint: the reason the Python stuff is so gnarly is because some people want a brewed Python and some people want a system one. In some respects if we went with 3) we're pretty much forcing everyone to use the same Python version which may reduce the support burden. I don't think that potential is a good enough reason personally but thought it was worth sharing here anyway.

[durka]

I agree with @jenshnielsen, the Principle of Least Surprise should prevail. In my mind, pouring a bottle shouldn't result in anything different than brewing from scratch, by default. So (2) or (4).

[no-name-party]

I dig 3. Because I don't use system python. But I also see the advantage in 2/4.

[daviewales]

I prefer 3 or 4, just because system Python requires you to set the PYTHONPATH, which I think is horrible. Although I like the principle of not replacing Apple stuff, I don't think system Python works particularly well with brew. I think the average user wants things to "Just Work™", and that can best be achieved by forcing brewed Python. If we could set site packages for system Python the way we do for brewed Python it wouldn't matter so much, but no-one wants to stuff around with PYTHONPATH.

Advanced users are the only people likely to care if brew installs its own Python, but they are also the only people who will understand how to set PYTHONPATH, and they won't mind using --build-from-source.

I used to use system Python with brew, and brewed Python for Python 3. However, I got sick of setting PYTHONPATH, so I put an alias for Python in my .bash_profile. As you can imagine, this stuffed up Python 3, so I then put an alias for Python 3 in my .bash_profile which set the PYTHONPATH for Python 3. (I know now I should have just made the Python 3 alias unset PYTHONPATH) However, this was all very messy and didn't work very well.

As soon as I switched to brewed Python, and got rid of all my PYTHONPATH rubbish, all my problems disappeared.

If we can't do this, I suppose we'll have to settle for 2.

[minrk]

system Python requires you to set the PYTHONPATH, which I think is horrible.

This isn't quite accurate, I've never used PYTHONPATH. I include homebrew site-packages by adding a homebrew.pth to ~/Library/Python/2.7/lib/python/site-packages/homebrew.pth (you can use /Library/ instead of ~/Library/ if you want it system-wide):

echo "$(brew --prefix)/lib/python2.7/site-packages" > ~/Library/Python/2.7/lib/python/site-packages/homebrew.pth

[mistydemeo]

My personal preference is 3. Using the Homebrew python is the path of least resistance, since it means there's no need to set either PYTHONPATH or a python.pth. Given that I see bottles as being a solution that is supposed to "just work" out of the box, it makes the most sense to me for python bindings and python-based software to just work without needing to set PYTHONPATH (or create a python.pth, or create more wrapper scripts for those things.

[GuillaumeDIDIER]

Just to be sure, we are not going to bottle for py3k (and for this one there is no system python, but either a python.org or brewed python).

[GuillaumeDIDIER]

Oh and by the way, is python itself bottled ? As if it is, bottling for brewed python will not cause a lot of overhead, though it will still force installing another python.
If we bottle for sys python, what about adding a caveat for the .pth ?
(Some other formula have that, mysql notably I think)

[iMichka]

Homebrew has the advantage to have all packages/libraries always up to date. Python is for me one of them, like the others. Look at other OS'es : I am currently using Fedora 20 with python 2.7.6 which ships by default, and when installing Python on Windows I chose to use packages from Christoph Gohlke's website, which are using python 2.7.6. In the Windows and Fedora cases, the packages are all built against Python 2.7.6.

I like the idea of "it just works" for novice users, and brewed python would be the way to go. In this case you (homebrew) manages the whole pipeline. I don't know why people would want to stay with the System's python. Most of the users don't care and would just have to install brewed python. Is there any reason to stay with the System's Python ?

So I'm voting for (3).

This leaves also the door open if you want to bottle for python 3 one day. This is easily manageable by the formula system : detect if brewed python 2 or 3 are installed, and pour the right bottle.

[MikeMcQuaid]

@minrk That's interesting, thanks. Think we should be recommending that instead of setting the PYTHONPATH?

A point in here that I've maybe understated: the PYTHONPATH stuff is only a problem for users that care about Python bindings i.e. are doing development outside of Python. If I'm installing e.g. ansible chances are I don't really care what language it is written it; I just want to use the thing. Given that libxml2 needs to have Python bindings enabled by default 3. or 4. are going to add a Python dependency to almost everyone using Homebrew. Give our goal on using system packages when possible and reducing the number of dependencies this seems like a disadvantage to me.

Thanks again for all the input, though.

[MikeMcQuaid]

@GuillaumeDIDIER Eventually the plan is to bottle everything.

[NikolausDemmel]

@MikeMcQuaid just to be clear: bottle everything still means 1 bottle per formula, even for the foreseeable future?

One more thing: It would be a big plus if there was a way to determine all the (installed) formulae that build python bindings (maybe there is already?), such that I know what to reinstall when switching from system to brewed python or the other way round. The wiki page on that says "make sure you reinstall all of them" but mentions just a few prominent ones.

[MikeMcQuaid]

@NikolausDemmel Yes, one bottle per-platform per-formula. Options should be used by the minority of users; if they are used by the majority they should become the default and be bottled. With the reinstalling Python bindings I'd suggest creating a separate issue about that. Thanks!

[daviewales]

@minrk That is much better than PYTHONPATH. However, it's still an annoying "extra thing" that users have to do in order to use the Python bindings. Most users will just install the program, then three months later try to use the Python bindings and find that although brew says Warning: program already installed, they can't seem to import the library into their Python programs. This will be confusing. The only non-confusing solution is to force brewed Pythons. It comes down to a battle between usability and the brew philosophy of not adding / forcing dupes.

That said, if we choose to continue with system Python, we should recommend @minrk's method, rather than telling people to set the PYTHONPATH. PYTHONPATH is evil, because it doesn't differentiate between Python 2.x and Python 3.x.

[GuillaumeDIDIER]

Nice piece of work.
So multiple bottles is a feature that will eventually come when the project gets even bigger.

[MikeMcQuaid]

@GuillaumeDIDIER It may. There's a bunch of stuff I need to do that I think is more important so not sure when (or if) I'll get round to bottling options.

[MikeMcQuaid]

So it turns out the "bottle against brewed Python, work with system Python" was a failure. There's too many formulae that store direct paths that we can't really rewrite properly. I'm going back to the "bottle against system Python, build from source against brewed Python" instead.

[mistydemeo]

There's too many formulae that store direct paths that we can't really rewrite properly.

There are also enough formulae, like Python itself, that install outside a keg and straight into site-packages, including Python's own pip and setuptools. :|

[MikeMcQuaid]

@mistydemeo Yeh. I guess we can post_install some of this stuff or see about putting it into the Cellar. Need to play around.

[MikeMcQuaid]

So, #28281 is the first issue indicating that this behaviour seems to be confusing. I think even if we improve the messaging there will be people that want to install bottles and have a Homebrew Python installed. The possible solutions (as far as I see):

1. Make Homebrew's Python keg-only so that it won't be in the user's PATH unless they explicitly set it.
2. Require an argument to be specified to build Python bindings against the Homebrew Python
3. Default all Python bindings to off in bottled non-Python formulae (i.e. disable in Boost, enabled in PyQt)
4. Something else?

[daviewales]

If we do 1, and the user explicitly sets the PATH to point to brewed Python, what happens then? Is it exactly the same as now (e.g. bindings built from source), or will it try to use the bottled system bindings?

My personal preferences is option 3, with a few extra details:

In the case of formulae such as boost which have optional Python bindings, and are primarily used by non-Python people, they should be bottled without Python bindings.

For packages such as PyQt which require Python bindings, they should be bottled with system Python.

If a user wants [other] Python bindings, they will have to build the entire package from source. Eventually, there will be bottles containing [other] Python bindings, but they do not exist at present. When [other] Python bindings are eventually bottled, it may be possible to bottle the Python bindings separately to the main package. This will allow people to install both Python and Python 3 bindings from bottles simultaneously. It will also allow users to install a package such as boost without Python bindings, then at a later date run brew install boost --with-python, and it will just grab the bottled Python bindings without needing to reinstall the entire package. (This is similar to Debian's apt-get install python-package)

[MikeMcQuaid]

If 1) it would be the same as currently, yep.

Yes, I think that sounds like the best approach: they default to off on most non-Pythony packages.

We're not going to adopt the Debian approach ever unfortunately.

[daviewales]

When you say "We're not going to adopt the Debian approach", do you mean we are not going to use python-package, or we are never going to provide individual bottles containing just the Python bindings?

[MikeMcQuaid]

Very likely never the prior, likely never the latter.

[MikeMcQuaid]

After a chat with @mistydemeo and thoughts form this thread I think I have a plan:

1. Stuff that just uses Python at runtime (and no-one really cares it is written in Python) should use the system Python e.g scons or ansible
2. Stuff that has non-essential Python bindings should be :optional rather than :recommended or always-on e.g. boost
3. Stuff that has essential Python bindings should be bottled against the Homebrew Python (or can be built-from-source against the system or external Pythons) e.g. pyqt

Thoughts?

[mistydemeo]

👍

[daviewales]

1. Makes sense.
2. Makes sense.
3. I am probably forgetting something here. What is the reason for bottling against Homebrew Python rather than system Python. Is it so we can avoid needing to set the PATH? (I don't have anything against this. I am just unsure what the logic is.)

Overall thoughts:
Sounds good. Go for it.

[MikeMcQuaid]

@daviewales With 3. it's a combination of not setting the PATH and from anecdata it seems that most people writing software with Python e.g. using stuff like PyQt all seem to use the Homebrew Python.

[iMichka]

1, 2, 3 seems a nice way to do it.

Maybe have a caveat or a message for each formula, telling the user against which python the build was made ? (Don't know if this is needed but this way the user knows what is being done :))

[NikolausDemmel]

It does seem like a reasonable compromise.

@MikeMcQuaid: Can I use the bottles for 1 even if use brewed python? I.e. those do not have any bindings that would be installed, right? An executing the binaries also chooses the right python (i.e. system)?

[MikeMcQuaid]

@NikolausDemmel Yes, the bottles just won't use the Homebrew Python.

[MikeMcQuaid]

@iMichka Will probably add a caveat and/or message for 3 and maybe 2.