Modifications to support iOS as a cross-compilation target

[freakboy3742]

BPO	23670
Nosy	@doko42, @ronaldoussoren, @ncoghlan, @ned-deily, @bitdancer, @freakboy3742, @rovitotv, @Lukasa, @rkuska, @mpaolini, @moreati, @controversial, @Mariatta
Files	20150314.diff: Patch for Python 3.4.2 source tree 20150426.diff: Updated patch for Python 3.4.2 source tree 20150504.diff: Updated patch with support for running the test suite. 20150628.diff: Updated patch, passing test suite on iOS simulator. 20150704.diff: Update 5 - only failing ctypes-related tests on device. 20151221.diff: Update 6 - single source tree for libffi_ios 20160301-brokentests.diff: Incomplete rebase of Update 6 against Python 3.6dev 20160217.diff: Patch against 3.5.1

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = None
created_at = <Date 2015-03-15.06:25:47.056>
labels = ['type-feature', '3.7', 'build']
title = 'Modifications to support iOS as a cross-compilation target'
updated_at = <Date 2019-09-04.00:08:40.855>
user = 'https://github.com/freakboy3742'

bugs.python.org fields:

activity = <Date 2019-09-04.00:08:40.855>
actor = 'josh.r'
assignee = 'ned.deily'
closed = False
closed_date = None
closer = None
components = ['Build', 'Cross-Build']
creation = <Date 2015-03-15.06:25:47.056>
creator = 'freakboy3742'
dependencies = []
files = ['38493', '39207', '39286', '39824', '39862', '41376', '42059', '42060']
hgrepos = ['300']
issue_num = 23670
keywords = ['patch']
message_count = 28.0
messages = ['238125', '238127', '238129', '238133', '238134', '238138', '238156', '238167', '238190', '238191', '238197', '238211', '240307', '241187', '241189', '242040', '242547', '245903', '246266', '256780', '256797', '261093', '261095', '267030', '267064', '267092', '284441', '305628']
nosy_count = 15.0
nosy_names = ['doko', 'ronaldoussoren', 'ncoghlan', 'ned.deily', 'r.david.murray', 'cvrebert', 'freakboy3742', 'Todd.Rovito', 'bkabrda', 'Lukasa', 'rkuska', 'mpaolini', 'Alex.Willmer', 'Luke Taylor', 'Mariatta']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23670'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[freakboy3742]

Proposal: iOS should be a supported platform for Python development.

The attached patch is a first pass at a patch to achieve this. It is a single patch against Python 3.4.2 sources, and requires no pre- or post-configure modifications.

Supporting iOS requires multiple builds - one for each of the hardware platforms that iOS supports (x86_64 simulator, ARMv7 and ARM64). These separate builds must then be merged into a single "fat" framework. The patch contains an iOS directory with a "meta" Makefile that manages this build-and-merge process. See the README in the iOS directory for details on usage.

The patch also introduces a new 'ios' platform type.

A sample XCode project for an iOS app is also provided as part of the patch.

iOS/README contains a couple of other notes about the build and the approach taken.

There are some known problems/limitations with this patch:

• It's a patch against 3.4.2, not hg trunk

• The code doesn't currently compile for ARMv7. In order to support ARM64, it has been necessary to use an unreleased trunk version of libffi; however, this version is currently broken for ARMv7. Older versions of libffi (including the formal 3.2.1 release) *do* work.

• The patch doesn't currently provide any way to run the test suite on the target platform. Testing is currently based on a simple smoke test of some basic features.

So - the patch isn't ready for commit to trunk. I'm presenting it for the purposes of getting feedback on the broad approach while I continue to resolve the known issues.

[ncoghlan]

Russell, just checking before I change the issue title: do you mean supporting iOS as a cross-compilation target? "Development" is ambiguous here, as you could mean development *of* CPython, rather than *in* Python, and I assume development *for* iOS largely takes place on x86_64 Mac OS X, Windows and Linux systems.

Slavek, Robert - assuming my understanding of the change is correct, I think would be a very interesting CPython enhancement from a Fedora Workstation perspective.

[ned-deily]

As a cross-compilation target. From a first quick look at it, it appears the patch requires a current Mac OS X system to build for iOS; the necessary standard build tools and SDKs for iOS are only available on OS X. These are the same build tools used for OS X builds. We already support universal builds for multiple architectures on OS X directly in one pass; unfortunately, the iOS builds require two different SDKs, one for running on the OS X-based simulator and one for the native platform archs which, I assume, is why Russell has gone for the separate builds for each arch and lipo-ed them together. I don't have an opinion yet about the use of the Setup.local configurations rather than modifying setup.py. I appreciate trying to keep the changes for a patch like this as isolated as possible. But, long term, that might not be the best approach assuming there is eventually agreement to fully support iOS as a standard platform (via cross-compilation). The bulk of the patch is the new version of libffi; presumably that will eventually be released upstream in the standard version of libffi so that having a separate copy wouldn't be required? In any case, as a work-in-progress, the patch is certainly nicely done.

[freakboy3742]

Nick: you are correct - these are changes to support iOS as a cross-compilation target, so you can run your Python code on an iOS device. Apologies for the confusion in nomenclature.

Ned: You're correct that the build needs to be run on an OS/X machine, and needs to be done in multiple passes because of the need for multiple SDKs.

I've taken the approach of using Setup.local for two reasons.

Firstly, iOS requires a "fully embedded" Python, because Apple doesn't approve of the use of dynamic loading in apps. Setup.local was the best way I found to force modules to be statically linked into the main library.

Secondly, because of the nature of the end-deployment, I imagine it's inevitable that serious end-users will want to customise their Python build for specific applications. AppStore submission rules prohibit running code that wasn't shipped with the app (i.e., Apple won't let you do an end-run around their app approval processes), so the set of modules you're actually using will be a known quantity. Since a "full" iOS Python build is 34MB or so, much of that bulk will be for binary support of modules that you aren't using, providing an easily configurable way to prune out unused modules from the built product seemed like a useful facility. There's also the issue of providing support for the standard library modules (like SSL) that have dependencies that can't be easily provided or specified out-of-the-box.

If there's a better/more palatable approach than using Setup.local, I'm happy to look into it.

The libffi code is a direct copy from the current (as of a week or so ago) libffi trunk. There have been some bug fixes applied since 3.2.1 was released in november that are necessary to support ctypes on ARM64 (but, as I flagged in my original comment, introduces bugs for ARMv7). The libffi repo suggests the next version will be 4.0, but I have no idea when that will be finalised. I'm currently trying to get their release plans confirmed.

As with OS/X libffi support, the source code needs to be generated, rather than just "./configure && make"'d. That said, the libffi_darwin code could be re-used, as the x86_64 components are the same for Darwin as the iOS simulator.

[ned-deily]

Ack, much to consider. Adding doko as I believe he has been most closely following developments with libffi releases.

[ned-deily]

A thought, primarily as a note to myself for further investigation: the current OS X framework build support and, to a lesser extent, universal build support have added a fair amount of special-case cruft to the build process, primarily in configure.ac and the main Makefile. Over the years, the build tools have evolved and gotten smarter (and more general) about SDK support and arch support using xcrun, as they were generalized to support iOS vs OS X builds. But, if it is necessary to do separate compile/link steps to support iOS framework builds because of the multiple SDKs, it *might* make sense to integrate those into the main Makefile and to isolate the framework build steps (for both OS X and iOS) to be more of an add-on step. The potential positive impact might be to simplify the current main Makefile and configure.ac by centralizing much of the framework build specific steps scattered throughout the Makefile *and* it could also simplify and speed up the iOS builds by not duplicating the platform-independent build steps. It might even be a useful abstraction for other non-Apple multiple-arch or cross-compilation builds.

Another factor to consider is support for building third-party extension modules standalone as opposed to building as part of the interpreter itself. It would be nice to be able to just use the traditional Distutils-based process for third-party extension modules. For OS X universal builds, we get that essentially for free because of the tool chain support for multiple archs. To do it automatically across multiple SDKs would require adding smarts to Distutils to do multiple compile/links and lipo merges.

[bitdancer]

Also remember that being a supported platform requires buildbots. If I'm reading this right that means two: one for the simulator build, and one that actually runs on the device (once the run-the-test issues is solved). I'm not sure what would need to be done to buildbot to support the scenario of building on one platform and running the tests on another after copying the binary, but I'm sure there must be some way to do it...buildbot is written in Python after all :) And I'm sure that capability will be useful for more than just iOS...with Android we can probably run buildbot on the device eventually, but being able to test a cross compile build there as well will be helpful (once there is one).

[freakboy3742]

Understood that buildbots are required. The subject has come up a couple of times on mobile-sig - however, I haven't got a good answer for exactly what this means in practice. Does build hardware need to be delivered to a specific build farm location, or can it just run in my home office? Does it need to be running 24/7, or just guarantee that at least X builds per week, or Y% of all builds are tested?

In terms of physical specifications - at a minimum, at least two iDevices will be required (an ARMv7 device, like a 5th Gen iPod, and an ARM64 device, like an iPhone 5S), tethered to a Mac that runs the simulator.

In terms of execution - there's definitely ways to remote-control the deployment and execution of apps, so I imagine buildbot integration is in the realm of the possible.

Android will have similar requirements, but Android also has an emulator which would allow testing without the existence of a physical device, which would provide an alternative approach.

[ncoghlan]

Rather than trying to do something ourselves on the mobile testing front, we may want to instead explore the free-for-open-source-project offering from Sauce Labs: https://saucelabs.com/opensauce/

Adapting that to run the CPython self-tests as an "app" might be interesting, but they're PyCon sponsors, so they may be amenable to helping to get that work properly if we need to ask for their help.

Getting that set up would also potentially be a useful tool for ensuring PSF web services are working correctly across multiple browsers and client devices. (I'd actually like to get us to the point of being able to do continuous deployment of those services some day)

[ncoghlan]

(Some work on the Buildbot site would still be required, but it would be triggering a Sauce Labs run rather than managing the mobile hardware directly)

[freakboy3742]

Nick: Finding a way to get on-device test results is next on my TODO list, once I've got the patch up to date for trunk.

FYI - Updating to trunk is currently blocking on bpo-22625 (which was introduced by the fix for bpo-22359).

I had thought about using a mobile testing farm "as a service", but I didn't know Sauce Labs had one. I'll keep that in mind when I get around to working on the test suite.

[bitdancer]

The Sauce labs option may render this obsolete, but to answer your question, Russell: our buildbot fleet is distributed in many people's offices, basements, and occasionally colo racks. The buildbot needs to be on-line 24/7, but it "calls out" and does not need any inbound ports open. See https://wiki.python.org/moin/BuildBot for the currently available documentation about setting up a buildbot for our fleet.

[ned-deily]

Russell, we talked a bit about this topic at the Python Language Summit today (after watching your excellent video). I think there was general agreement there that having iOS support would be "a good thing" along with the recognition that the requirements for support of a mobile platform like iOS differ in some significant ways from most traditional platforms that Python has supported. I agreed to shepherd this issue. I have some initial thoughts but will write up something more detailed shortly.

[doko42]

not reviewing everything, but

• libffi_ios* will be bitrot. I understand that you'll
  need an unreleased version, however I would much
  prefer that any additional patches are added in the
  repo, and maybe then patching libffi from the iOS build
  system. This way the code will see updates when libffi
  is updated.

• I don't like the copies of Setup.*. Again this will
  bitrot. Please see
  http://bazaar.launchpad.net/~doko/python/pkg3.4-debian/view/head:/rules
  search for "define __post_configure"
  Basically this patches Setup.dist to build a bunch
  of extensions statically. Can you use something
  similar, or improve that?

• the change to setup.py might deserve a comment.

• are constants in plat-ios really the same for every
  architecture?

• how is the gnu triplet / quadruplet called for iOS?

[freakboy3742]

Hi Matthias:

• The libffi situation on iOS is much the same as it is on OS/X - it needs to be pre-generated. This can't be driven by the existing build system - libffi's Makefile requires a separate pre-generation step. There's certainly potential for reuse between the OS/X and iOS generated files, though - there's no reason I couldn't merge the libffi_ios tree into the libffi_darwin tree - they reuse a lot of the same code for the x86-64 platform.

• Regarding Setup.* - if shipping specific Setup.* files is a problem, I can certainly look at enhancing the makesetup process to generate these, rather than shipping generated files.

• I'll add a comment to the setup.py change. Essentially, it's an edge case of the current setup.py - if you don't have *any* external modules, the current logic breaks evaluating the size of the columns to use when displaying external modules that haven't been compiled - although it's displaying the list of *missing* modules, it uses the longest name in the *existent* extensions to determine the display column width.

• The Plat_ios constants certainly appear to be the same for all platforms; admittedly, though, I haven't done a thorough audit of this. I'll put this on my todo list.

• The build triples are "armv7-apple-ios" for older 32 bit iOS devices, "aarch64-apple-ios" for newer 64 bit devices, and "x86_64-apple-ios" on the simulator. Builds for "armv6-apple-ios" and "armv7s-apple-ios" are also plausible, but armv7 and aarch64 is what XCode supports as a default configuration at present.

[freakboy3742]

Here's an updated patch that integrates some of the feedback that has been received so far.

Notable changes:

• The code now works for ARMv7. Unfortunately, the price for this is a new libffi_ios_aarch source directory, containing generated source tree for the ARM64 platform. I've been working with the libffi community to address the compilation problems that exist in trunk, but so far, this hasn't resulted in a fix for the problems that have been introduced in their trunk code.

• I looked into merging the libffi_ios and libffi_osx directories, but this proved to be harder I thought because of legacy PowerPC support. libffi still ships PowerPC sources, but they're no longer included in OS/X builds, because Apple stopped supporting PowerPC with OS/X 10.6 (released in 2009; the last PowerPC Apple hardware was shipped in 2006). As a result, it's difficult to find compilation instructions, and even harder to find actual hardware to test builds. If support for PowerPC architectures (on OS/X) was deprecated, this would make merging the libffi_ios and libffi_osx a trivial activity.

• I've audited plat_ios/IN.py, and can now confirm that it *is* identical between platforms.

• I have looked into replacing Setup.* with some sort of post-configure procedure as suggested by @Doko. Unfortunately, it's not that simple. The problem is that there is already 2 places where the build requirements for a built-in module are defined. Modules/Setup contains one set of instructions (although those instructions are often commented out). The second set is hard coded into setup.py.

The versions in setup.py are probably more reliable (as they're the ones actually used for most platforms), but you need to have a working Python to access them. However, at the "post-configure" point, you don't have a working Python, so there's a bootstrapping problem.

I've refactored the Setup.ios-* definitions so that there's less duplication. There's now a Setup.embedded that contains the common build instructions for all the modules without platform-specific build instructions. Setup.ios-* just contains the platform-specific build instructions (in much the same way as setup.py has configure_ctypes_darwin).

To work around this completely, we'd need to either introduce the need for a bootstrap Python so that a post-configure setup could access the build instructions contained in setup.py, or massively refactor the process of building modules so that there was a set of build instructions that both makesetup and setup.py could use.

[freakboy3742]

This new patch (20150504.diff) adds support for running the Python test suite The new patch is standalone, and contains everything in the previous patch.

An XCode project (Tools/iOS-test) has been added to the source tree; this project contains bootstrap C code to start and run the full Python regression test suite.

There's also a new target in the iOS/Makefile meta-buildfile - "make test" - which will compile a debug version of the Python framework, and install it into the iOS-test project.

Getting the test suite to run has revealed one major limitation of the iOS platform - system calls like fork, exec*, and spawn* don't work. The OS calls *exist* at an API level - but if you use them, they either crash, or they lock up the device while you wait for a subprocess that will never execute. This makes sense considering the platform itself - there's no such thing as a "background process" in iOS; background tasks are very heavily sandboxed.

A number of other minor problems have been identified as a result of running the full test suite; they have been addressed in the patch.

The test suite still has 5 failures on the simulator. I'm investigating the cause of these failures.

There's a couple of more failures on a physical device - 13 failures in total. These device-specific failures appear to be largely due to ctypes problems and a permissions issue with os.mkdir.

If you run the test suite in XCode as a debug binary, the debugger will stop whenever a SIGPIPE, SIGINT, SIGXFSZ, SIGUSR1 or SIGUSR2 signal is raised. You can just hit continue in the debugger, and the test will continue. To work around this, you need to run the following debugger commands before the first signal is raised:

process handle SIGPIPE -n true -p true -s false
process handle SIGINT -n true -p true -s false
process handle SIGXFSZ -n true -p true -s false
process handle SIGUSR1 -n true -p true -s false
process handle SIGUSR2 -n true -p true -s false

I've been doing this by setting a breakpoint in the main.c method; I'm investigating other ways to automate this.

[freakboy3742]

Another patch update - the code now passes the full Python test suite on the iOS simulator.

There are still a couple of failures on device; as before, these appear to be due to ctypes problems and a permissions issue with os.mkdir.

[freakboy3742]

Another update - this time, there are only 4 failing tests on device, all related to ctypes issues.

The sample Xcode project and iOS-test harness have been modified, simplifying the project layout, and using Apple-preferred directories for resources when deployed to device.

[freakboy3742]

Another update - the issue with libffi has been resolved, so there is now only need for a single copy of libffi_ios sources.

This directory could potentially be merged with the libffi_osx sources, except that libffi no longer supports PowerPC as a compilation target for OS/X builds (and neither does Apple). If PowerPC can be dropped from the list of supported platforms, libffi for OS X can be updated to the current trunk sources.

[ncoghlan]

Mac OS X 10.5 was the last version to support PowerPC architectures, and according to https://en.wikipedia.org/wiki/Mac_OS_X_Leopard#Release_history, the final version of that was 10.5.8, released in August 2009.

While we don't have a section in https://www.python.org/dev/peps/pep-0011/ spelling out our support policy for Mac OS X the way we do for Windows, Ned is still making dual architecture binaries available for 2.7.x and 3.5.x as seen at https://www.python.org/downloads/release/python-2711/ and https://www.python.org/downloads/release/python-351/

[moreati]

I've done my best to rebase Freakboy's patch onto 3.6-dev. The attached applies cleanly, but the testsuite goes into an infinite loop. It's a start at least. At a guess the problem is in Lib/test/libregrtest/ or Lib/test/regrtest.py where the patch changes sys.exit() calls to return statements.

I took the liberty of filtering out whitespace & other cosmetic changes that were in 20151221.diff.

[freakboy3742]

Alex - The usual cause for that problem isn't regrtest (at least, not directly). The cause is one of the tests in the suite spawning a subprocess. Due to the way the test harness works, when the test suite forks/spawns a subprocess (e.g., to run a network server or a multiprocess worker), the fork causes the a second version of the test suite to start, rather than starting the server subprocess.

I've attached a patch for 3.5.1. This includes a patch to work around bpo-22625; the patch is definitely suboptimal, but it lets a build come to completion. What is really needed (as requested on python-dev) is some help working out how to fix bpo-22625 properly.

[controversial]

Are you aware of Pythonista? I have no affiliation, but I'm a fan of the app and the community surrounding it. See http://omz-software.com/pythonista/ for details. I'm sure communication with the developer of the app could yield some useful insights.

[freakboy3742]

Yes - I'm aware of Pythonista; the author of that app contracted me to develop the 3.5 patch that is attached to this ticket :-)

[controversial]

Ok, great!

On Fri, Jun 3, 2016 at 7:34 AM Luke Taylor <luke@deentaylor.com> wrote:

Ah, cool!
On Fri, Jun 3, 2016 at 2:58 AM Russell Keith-Magee <report@bugs.python.org>
wrote:

>
> Russell Keith-Magee added the comment:
>
> Yes - I'm aware of Pythonista; the author of that app contracted me to
> develop the 3.5 patch that is attached to this ticket :-)
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23670\>
> _______________________________________
>

[rovitotv]

Russell, this is excellent work!!!! I am truly amazed that within a couple of hours I had Python built and running for the iOS simulator this is a feat I didn't think was possible. Perspective on me I am very familiar with Linux, C, and Python but know very little about iOS and XCode keep that in mind when reviewing my comments.

MacOS: 10.12.2
Xcode: 8.2.1 (8C1002)
iOS: 10.2 (14C89)
Python source code: 3.5.1

I applied your last patch file name 20160217.diff. The first issue I ran into was clock_settime() is unavailable in iOS so I went into timemodule.c line 164 and commented out the line:
ret = clock_settime((clocked_t)clk_id, &tp);
then added this line
ret = 0;

Next I think a more serious problem was with Python/random.c line 92 "Python/random.c:92:19: error: implicit declaration of function 'getentropy' is invalid in C99 [-Werror,-Wimplicit-function-declaration]"

So I did the same thing as above commented out the lines with the getentropy function and added a line of res = 0. Who needs entropy anyway? I wonder if this problem is related to bpo-29057 (http://bugs.python.org/issue29057).

Next in your README file you list a file called main.c when the actual code is main.m.

Next main.m was not compiling with Xcode so I replaced with this line after looking at your Python-iOS-template project on GitHub, as I am not qualified to make these types of changes:

-line 31 was not compiling so I replaced:
wpython_home = Py_DecodeLocale([python_home UTF8String], NULL);

-line 54 had a similar error so I replaced:
python_argv[0] = Py_DecodeLocale(main_script, NULL);

-line 56 had a similar error so I replaced with this line:
python_argv[i] = Py_DecodeLocale(argv[i], NULL);

-line 89 comment it out to remove the assertion failure, I am not sure what
this line is supposed to do but no assertion failure seems like a better result.

I didn't want to blindly attach a diff as I don't understand the Apple lingo but I wanted to provide feedback in the hope that it helps get this patch committed so iOS becomes a target for CPython in the near future. Thanks!

[freakboy3742]

For those interested, I've started tracking these patches on Github: https://github.com/freakboy3742/cpython

The 3.4, 3.5 and 3.6 branches are tested and compile as of the time of this comment; the master branch has also been updated, in preparation for the release of 3.7.

I haven't submitted pull requests because there are still some issues that Ned raised in discussions at PyCon US.

[ned-deily]

After discussion with @freakboy3742, we agreed that much has changed since this issue was opened and so it makes most sense to close this issue in favor of a new one, with PRs targeted for 3.13, that reflect the current approaches and changes needed in cpython, for the most part, simpler changes.