Support earlier OS X versions in wheels

[minrk]

Wheels built for 10.6 (e.g. Python.org) work on later targets (e.g. System Python). The reverse is not supported.

[minrk]

This matches the longstanding behavior of easy_install, which supports installation of eggs built for 10.X on 10.Y, as long as Y ≥ X.

[dstufft]

I'm not specifically aware of the implications of doing this. I'm going to tag this for the 1.6 milestone as we should figure it out and either accept or reject for 1.6.

[minrk]

I'm not specifically aware of the implications of doing this.

I'm not either, but I have been building eggs and wheels with Python.org (targets 10.6) that work on System Python (10.7-10.9) for years, and have yet to hear of a failure. Looking around, this seems to be relatively common.

If there is a way that I can specify 'this works on 10.6-10.9', or better yet '>= 10.6' explicitly, that would be a fine solution, but uploading multiple copies of the same wheel with different filenames is no good.

[minrk]

This PR now fixes both regressions due to the newly strict equality test. Multi-arch (intel) wheels will install on single-arch (x86_64), since they necessarily include the right code. This PR now brings back the status quo of Python.org-built wheels (10.6-intel) being installable on just about any reasonable recent Python.

Where is the best place to have the discussion about whether this is the right way to go or not?

[ChrisBarker-NOAA]

This should be discussed on distutils list. But also maybe the python-mac list, as that is were we are mostly likely to find the technical expertise to let us know if this is dangerous.

Also is "the status quo of Python.org-built wheels (10.6-intel) being installable on just about any reasonable recent Python" My experience with binary eggs was that they NEVER installed (into a universal python) without name mangling.

But if it is supported by Apple to link in libs built against older SDKS that the host application was built for, then this is pretty cool. And your experience seems to indicate that it is supported.

[minrk]

My experience with binary eggs was that they NEVER installed

System Python (macosx-10.9-intel) definitely installs eggs built by Python.org (macosx-10.6-intel) (to confirm, just do easy_install pyzmq in a virtualenv with /usr/bin/python). The comparison is in pkg_resources.compatible_platforms.

I was wrong about the arch, though - easy_install does strict comparison of the arch (intel will not install on x86_64, even though it includes x86_64 by definition). I don't know where I got that idea. That being the case, I can split these two proposals into separate PRs.

But if it is supported by Apple to link in libs built against older SDKS that the host application was built for, then this is pretty cool.

From the Apple documentation:

Choose a deployment target. This identifies the earliest OS version on which your software can run.

Which seems pretty clear to me that >= is the right comparison, not ==.

[ChrisBarker-NOAA]

"""
From the Apple documentation:

Choose a deployment target. This identifies the earliest OS version on which your software can run.

Which seems pretty clear to me that >= is the right comparison, not ==.
"""

Except that those docs are talking about building an "application" -- no mention I can see of building a library, and what that built library can be linked against. Maybe I'm just being paranoid, but it seems this could be unstable.

Using otool -L on my system python, I see it is lined to:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

while an arbitrary dylib in the python.org build is linked to:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)

so same "compatibility version", different "current version".

If "compatibility version" means what is sounds like it means then I guess were good. Though I'd still like to read it somewhere that that's how this works...

[ned-deily]

It would be good to fully support OS X universal archs when selecting binary distributions. The complete list of universal arch names known to Python Distutils is shown here: http://hg.python.org/cpython/file/default/Lib/_osx_support.py#l463
e.g. "fat", "intel", "fat3", "fat64", and "universal". It would be best if all of them were supported in pip.

The version number that appears in Distutils-supplied names, e.g. "-10.6", is actually the value of the system ABI ("MACOSX_DEPLOYMENT_TARGET" or "-mmacosx-version-min="). Because of its strong preference for dynamic linking to system libraries and frameworks, Apple tries very hard to preserve binary compatibility so that something linked with an OS X release n will continue to work on at least n+1 and, usually, for n + many more than 1. In general, then, a test for selecting binary distributions for OS X should be that the ABI value in the Distutils-supplied name should be greater than or equal to the ABI value of the Python interpreter to which it is being installed. That value is available by using sysconfig:

sysconfig.get_config_var("MACOSX_DEPLOYMENT_TARGET")

In the specific case of taking C extensions built for OS X using a python.org 2.7.x 64-/32- python and installing/using them with an Apple-supplied 2.7 system python on OS X 10.6 through 10.9, AFAIK this seems to usually work OK. It's neither officially tested (by us) nor claimed to always work. (We do test that extensions built with a python.org Python targeted for 10.6 work with that same python.org Python on 10.6, 10.7, 10.8, and 10.9.) But it does usually because the system Python 2.7 and python.org Python 2.7.x are built with mostly compatible options, the most important being that both use a narrow Unicode build ("ucs2" for Python 2). In general, prior to Python 3.2 and PEP 3149 (version tagged .so files), there is no easy way to tell by inspection with what Unicode ABI a binary distribution was built. Some distributors, e,g, MacPorts, provide the option for wide Unicode builds for Python 2 and Python 3 (prior to Python 3.3 where the Flexible String Representation removes the need for Unicode build options). Using an extension built with a UCS2 Python on a UCS4 Python would likely fail in unpredictable ways. This is an open issue for wheels and one that I believe is to be fully covered with Metadata 2.0.

Another potential gotcha is for any extension using C++ code. On OS X 10.9 Mavericks, the default C++ runtime is now libc++; previous it was libstdc++ which is still available on 10.9 for binary compatibility. Problems can arise when mixing C++ routines that use the two different runtimes. I don't know that there is anything we can do to mitigate that. But I bring it up in case any of the scientific routines use C++. I'm sure there are other subtle potential gotchas; the questions are how likely are any users to run into them and how to mitigate them. At this point from a usability point of view, the best course may be to assume everything works (outside of the known issues) until proven otherwise.

[minrk]

pyzmq wheels include libzmq as a Python extension, which is a C++ library. I will try to carefully investigate any compatibility there. A quick test:

	Python.org (10.6-intel)	System Python (10.9-intel)	Brew Python (10.9-x86_64)
Compiler	gcc-4.2	clang	clang
libc++	/usr/lib/libstdc++.6.dylib	/usr/lib/libc++.1.dylib	/usr/lib/libc++.1.dylib

And on a test (after applying the patch in this PR, so pip doesn't prevent installation), it seems to work as I expect:

• Python.org wheel works on all three Pythons
• System Python wheel works on Homebrew

But obviously, my machine is set up for development, so there may be some different assumptions for a fresh install or the typical user. I am happy to set up and test with VMs, if there are any more specific cases people would like to investigate.

[ned-deily]

I don't claim to be an expert on this but I think there would be a potential problem if someone were to install a mixture of C++ extensions, some linked with libstc++ and some with libc++. As long as they all use the same runtime, I guess there shouldn't be any problems. Python itself (at least the python.org one) has no C++ dependencies.

[ChrisBarker-NOAA]

Thanks Ned.

I do know that the new c++ lib with 10.9 has been a pain for people trying to build stuff.

But in this case, we are primarily talking about extensions built on older systems being run with newer python builds -- I guess the C++ issue is moot because Python itself doesn't use C++.

When I use ottol on a python extension built with teh python,org python on my 10.7 sytesm, I see:

/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)

if I built this on 10.9 (which I can't, because I dont have 0.9, and because apparently this lib is incompatible with this code anyway -- another issue) would it use libc++ instead? And would that be a problem if yu had two extensions running in the same pyton, one with each lib?

But I agree with Ned here -- let's assume it works and deal with it if it doesn't...

Also -- for the macports Unicode build issue -- I don't think we need to support binaries for Macports or Homebrew on PyPi -- those systems are DESIGNED to build stuff for themselves...

-Chris

[ChrisBarker-NOAA]

Min -- it looks like if you build a C++ extension on 10.9, it's linked to libc++ -- so I expect it wont' run on an older system at all -- have you tried that? If you point me to binareis, I'd be glad to test.

-Chris

[ned-deily]

Apple clang has a "-stdlib=" option that allows you to specify which runtime to link with. You might be able to override that in Distutils by carefully using an LDSHARED environment variable.

[minrk]

I tried to contrive a case with a mixture of C++ libs:

• C++ library compiled with gcc-4.2 (stdc++)
• Python bindings of the same library compiled with clang (libc++)

And it all seems to work. I even reversed it, with no apparent issue. I don't know exactly what sort of failure I should expect, but at least it doesn't appear to be the case that every single potential mixture of libc++ and libstdc++ fails catastrophically.

[ned-deily]

I know that the MacPorts project has run into issues with mixtures of the two runtimes. They have a FAQ on the subject here: https://trac.macports.org/wiki/FAQ#libcpp

[minrk]

it looks like if you build a C++ extension on 10.9, it's linked to libc++ -- so I expect it wont' run on an older system at all -- have you tried that?

It depends on the Python (and technically the compiler environment variables at build time). With no environment config, Python.org uses gcc/g++-4.2, which is not clang, and links stdc++ even when run on 10.9.

A wheel built on 10.9 with Python.org 2.7.6: https://www.dropbox.com/sh/hbyd6icb3kskg5h/emky1ixcZC/pyzmq-14.1.0_rc1-cp27-none-macosx_10_6_intel.whl
And 10.9 System Python (you might rename this wheel, and see if it works on older System Python): https://www.dropbox.com/sh/hbyd6icb3kskg5h/4lJuUThIIZ/pyzmq-14.1.0_rc1-cp27-none-macosx_10_9_intel.whl

[minrk]

All this said, I think this sort of linking question is out of scope of the pip/wheel tag format. You can have the exact same sort of problem with MSVC versions on Windows, but PEP-425 doesn't try to express that.

[ned-deily]

Yes, the more detailed questions of ABI compatibility were deferred to the Metadata 2.0 PEPs (http://legacy.python.org/dev/peps/pep-0426/). There are also similar issues for the various Linux distributions. As I recall, the assumption was that Windows users would be most likely to be using python.org Pythons, as there are no system Pythons there, and also would be less likely to have a development environment installed, so it was reasonable to only have to target to one particular binary ABI per Windows Python version, unlike on the other major platforms.

[minrk]

@ned-deily I added the missing multi-arch tags, thanks.

[ned-deily]

Excellent, thanks!

[dstufft]

Yea we don't need the system to be bullet proof. Generally what we shoot for at the moment is "given the selectors available is it reasonable to expect that we can select a Wheel that will work". Corner cases and such not excluded. For some projects, at least right now, not publishing a Wheel may very well be a reasonable decision if they need something that makes binary distribution difficult. Y'all have been way more thorough then I personally expected :)

@ned-deily I'm not sure in what cases the universal, fat, etc would show up. Right now Wheels are just created using the value of distutils.util.get_platform(). If more information makes sense on OSX then we'll need to amend PEP425.

[ChrisBarker-NOAA]

@dstufft :
Yes, this went a bit far -- but in fact, we're trying to do something similar (to Windows) here for OS-X users: binary wheels on PyPi should support the python.org builds. The issue is that if they also support the Apple system builds, then it would be nice if we could make that work, too.

The greater issues with Macports, Homebrew, etc, I think are best deferred, if ever supported at all.

[Ivoz]

The biggest problem I have with OS X' system python is that it uses OS X' system OpenSSL, which is deprecated and unmaintained. Also, they don't want to fix anything about it.

https://hynek.me/articles/apple-openssl-verification-surprises/

And a critical thing pip uses to download packages... is ssl connections.

Another issue with Python.org Pythons is simply unfortunate... installed where they are as a default OS X application, they often come across a lovely little bug where their path includes a space (often from the naming of a hard drive). Given that virtualenv uses hashbang scripts to select pythons, and every kernel I know about disallows spaces in hashbang paths...

Homebrew is luckily in that its default install location is without spaces, /usr/local/bin. It will also take over the $PATH's python, unlike Python.org Python, which is often a trap for beginners expecting to get the latter after they install it and use anything on the command line.

homebrew python has actually easily become the python of least resistance, in my experience. I'd rather not deferr supporting it.

[ronaldoussoren]

Ivoz: I don't understand your coment about spaces in file names w.r.t. the python.org installer. It install Python into a path without spaces (/Library/Frameworks/Python.framework). The only bits that do install into paths with spaces are the helper applications (e.g. /Applications/Python 3.3/IDLE.app), but that doesn't affect scripts.

Using Apple's version of openssl is unfortunate, but does give access to the CA list in the keychain, which is the primary reason we haven't replaced it yet. That is, with the system SSL you get CA verification while using another OpenSSL doesn't unless you also have a CA list in the filesystem (which Linux systems have by default, but OSX does't). See also http://bugs.python.org/issue17128

[rgommers]

So far in this discussion I didn't see the distinction made between compiling on an older OS X versus compiling on a newer one but using the SDK for the older version. The former is always safe in my experience, the latter much less so. For scipy releases I experimented with compiling with MACOSX_DEPLOYMENT_TARGET=10.3 (to match the python.org Python installers) on 10.6, and this would give strange crashes.

I'm not sure if it's possible to make a difference between OS and SDK numbers. If not, you could decide to still enable installing with >= instead of == but leave it up to packagers to not use SDK if those are problematic.

[ChrisBarker-NOAA]

On Sat, Mar 8, 2014 at 2:17 AM, Ralf Gommers notifications@github.comwrote:

So far in this discussion I didn't see the distinction made between
compiling on an older OS X versus compiling on a newer one but using the
SDK for the older version. The former is always safe in my experience, the
latter much less so.

Well, it theory, you can compile for older systems on newer ones, though
you are quite right - it can be a real trick, and hard to test (for the
developer with out an older system handy), of course. So the easiest/safest
think is to compile on the oldest system you want to support.

But it looks like Min and Matthew have figured it out, anyway.

For scipy releases I experimented with compiling with

MACOSX_DEPLOYMENT_TARGET=10.3 (to match the python.org Python installers)
on 10.6, and this would give strange crashes.

yup -- it's hard.

I'm not sure if it's possible to make a difference between OS and SDK
numbers. If not, you could decide to still enable installing with >=instead of
== but leave it up to packagers to not use SDK if those are problematic.

I think his is a "how" issue, rather than a "what" issue -- there are a
myriad of ways to compile a package so that it won't run properly on the
targeted systems, this is only one of them -- we seem to have converged on
the what: "packages on PyPi should be built to run on the same OS versions
as the python.org builds, and pip should allow them to be installed into
newer python builds.

-Chris

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

[rgommers]

"packages on PyPi should be built to run on the same OS versions as the 
 python.org builds, and pip should allow them to be installed into newer python builds."

yes, fully agree.

Just wanted to point out the SDK issue specifically, because the python.org builds are done on such an old OS X version that most people won't have access to a machine running it anymore. But it's up to packagers to solve that; not directly related to pip.

[blowekamp]

I have run into this issue as well. I posted it about it here:
https://groups.google.com/forum/#!topic/python-virtualenv/tUdEE88hC_c

I have been happily building my binary eggs on 10.7 for python 2.7 and the users only get issue if they try to install then on a Canopy of Enthought distribution.

After working to get the eggs built, I was quite disappointed I could not install them as expected.

[minrk]

It sounds like there is agreement that this is the right way to go. Is there anything more I should do before it can be merged?

[minrk]

rebased

[rgommers]

@matthew-brett that stack testing looks fantastic!

[matthew-brett]

The testing is almost entirely code from @mrterry and https://github.com/matplotlib/mpl_mac_testing - thanks Matt.

[ogrisel]

The failure reported by travis looks like a false alarm (timeout) and not a real failure. Could someone with the travis rights relaunch the failing build to confirm?

[matthew-brett]

Min - can you trigger a rebuild?

[Ivoz]

fyi, #1856 needs to be resolved to stop the py32 stall

[minrk]

Only repo owners can request Travis re-run tests

[dstufft]

Thanks for your contribution, sorry this took so long to get done!

[minrk]

Any chance of this getting into a 1.6 or 1.5.7 release soon? With OS X 10.10 in the very near future, it would be nice to not need to remove and re-upload all wheels to add the macosx_10.10 tags.

[dstufft]

I'm planning to release 6.0 (previously it was 1.6.0) in the near future.

[minrk]

Awesome, thanks.

[ned-deily]

As a side note, this would only affect use with Pythons built on 10.10 (or more precisely with MACOSX_DEPLOYMENT_TARGET set to 10.10), so, presumably, the system Python 2.7 and locally built ones. It would not affect use with Pythons installed from python.org OS X binary installers; so existing wheels that work with those Pythons should continue to download and work with them on 10.10. (Edit: just to be clear, only the latest, current python.org releases are expected to work correctly on 10.10 when it is released, e.g. Python 2.7.8 and 3.4.2. Older releases will likely have problems in this area, primarily because of the move from one-digit to two-digit, 10.9 -> 10.10.)

[matthew-brett]

Right - the current set of wheels will only work for Python.org Python on OSX 10.10. They won't get picked up (with pip 1.5.6) for system Python, homebrew or macports for 10.10.

[ogrisel]

They won't get picked up (with pip 1.5.6) for system Python, homebrew or macports for 10.10.

Which is probably the majority of Python installations under OSX.

[ChrisBarker-NOAA]

I don't think it's the majority -- not everyone updates to the latest OS version, and macports and homebrew provide ways to compile stuff yourself, and some of us use python.org rather than Apple's (hardly should use Apple's as far as I"m concerned...). And teh pyton.org builds are the primary target.

All that being said -- it would be good to have it work!

[ned-deily]

Apple just announced that OS X 10.10 is available today so people will be looking for this.

[matthew-brett]

I don't think Olivier meant a majority of OSX users, only a majority of soon-to-be OSX 10.10 users.

I personally have no idea what the relative proportions of system / Python.org / Macports / Homebrew Python installations is - does anyone have any data?

@dstufft - are you planning to release 1.6.0 in the next week or so? If not I think we will have to go through renaming pypi wheels for numpy / scipy / matplotlib / sklearn etc.

[pfmoore]

pep425tags.py is also in the wheel sources. Does that need to be changed in line with this change? Maybe that file should be split out into a central project (packaging?) rather than being copied in the 2 projects?