Support Apple app store bitcode

[brson]

Bitcode is the future of Apple's app distribution and we don't support it now. Doing so is tricky because Apple choses whatever rev of LLVM they want and upgrade it per their whims. We can't couple our LLVM to theirs because we have our own needs and can't be forced to upgrade whenever Apple decides.

Here are some options:

• We ship an entirely separate toolchain for targetting Apple bitcode. This is quite ugly.
• We change rustc to dynamically load LLVM, create a rust-llvm package and an optional rust-llvm-apple package, and have rustc load the apple LLVM to deal with their bitcode.
• We create a rustc_llvm_apple crate and just link two entire LLVMs into rustc at all times. This probably would not fly
• We create a build of rustc called rustc-apple and ship it in an optional rustc-apple package. It is exactly like rustc except instead of linking to rustc_llvm it links to rustc_llvm_apple. When rustc gets a request to emit bitcode it just defers completely to the rustc-apple binary. I actually fancy this solution quite a bit.
• We could investigate a bitcode sanitizer to translate our bitcode to theirs. I think the likelihood of success here is low because maintenance. I believe RenderScript does this.

I think the LLVM dynamic loading and defer-to-alternate-rustc solutions are most promising.

cc https://users.rust-lang.org/t/ios-rust-integration/6928/4

cc @bluejekyll

[nagisa]

We change rustc to dynamically load LLVM, create a rust-llvm package and an optional rust-llvm-apple package, and have rustc load the apple LLVM to deal with their bitcode.

Dynamically loading LLVM is certainly not an option. The LLVM API is less stable than its bitcode and, if we build against LLVM x.y, then having LLVM x.z instead of x.y will almost certainly break rustc. Just link to the system LLVM statically (like --llvm-root already does).

We create a rustc_llvm_apple crate and just link two entire LLVMs into rustc at all times. This probably would not fly

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

We could investigate a bitcode sanitizer to translate our bitcode to theirs. I think the likelihood of success here is low because maintenance. I believe RenderScript does this.

Okay… so, the thing is that LLVM bitcode format is kinda stable between versions of LLVM. Does apple do something unusual to make that not true, or is the bitcode format used not LLVM altogether (i.e. has apple specific stuff)?

---

I feel like I heard all the way back in 2013-ish that using LLVM bitcode as distribution format is stupid. I certainly agree; does distributing native binary libraries not work anymore for iOS?

[nagisa]

I’m also very interested in what would happen when Apple upgrades to LLVM version with incompatible bitcode format internally and people still compile stuff with old compilers.

[bluejekyll]

I experimented briefly with this, i.e. output bitcode during cargo builds, then tried to create a static library with the bitcode included. When I tried to llvm-link against the bitcode library, I got an incompatible LLVM version from the bitcode.

I don't have an easy test to reproduce this, but my guess is there is just a dumb version check that denies linking between different LLVM versions? Or, I just did something entirely wrong. I will try to come up with a test case when I have some time to look at this again.

[sanxiyn]

It would be useful to have exact error message from llvm-link.

[nagisa]

@bluejekyll LLVM has a number of bitcode files in its test directory. Tests against these bitcode files are run continuously (llvm-dis-3.8/opt-3.8 understands a 3 year old bitcode file from 3.2 just fine, for example), so it must be some apple stuff.

[ricky26]

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

This is somewhat easier said than done - as far as I'm aware, the only blessed apple-llvm version for App Store uploads is the one shipped with the current Xcode. This also potentially means maintaining the LLVM bindings for two LLVM versions (not necessarily two neighbouring minor versions, either). I don't think it's valid to just use an old version of apple-llvm.

I’m also very interested in what would happen when Apple upgrades to LLVM version with incompatible bitcode format internally and people still compile stuff with old compilers.

I think they avoid this by only allowing the latest Xcode to submit apps (and I'm pretty sure it's burned into the output image which LLVM version you used).

IIRC, the bitcode is packaged for each architecture, since cross-architecture bitcode isn't generated by clang (I think that's an anti-goal of clang and bitcode in general). It's stored in a section for each object file so at the very least it's duplicated. That could be part of why someone mentioned that bitcode might not be the best way of going about this.

I feel a bit like all of the recommended solutions are somewhat icky. The least messy way I can think of is to allow different targets to overload the codegen behaviour and have the Apple codegen path be in a dynamic crate. (Which would just be the regular codegen path compiled against apple-llvm.)

Since this bug mentions the App Store, is it worth talking about the exception handling story here? (i.e. panic=abort is strictly required at the moment.)

[bluejekyll]

LLVM has a number of bitcode files in its test directory. Tests against these bitcode files are run continuously (llvm-dis-3.8/opt-3.8 understands a 3 year old bitcode file from 3.2 just fine, for example), so it must be some apple stuff.

@nagisa thanks for letting me know this. It gives me hope that there might still be a solution here, and that I was probably doing something wrong.

[nagisa]

@ricky26 good points.

as far as I'm aware, the only blessed apple-llvm version for App Store uploads is the one shipped with the current Xcode.

Isn’t Xcode LLVM on apple the same thing as system LLVM? I meant the Xcode LLVM then. We would have to make sure xcode is always the most recent version when shipping rustc trains.

Of course the way apple does things rules out us producing a valid apple bitcode with an old version of rustc and essentially forces us throw all the benefits our stability story provides out the window, and I don’t see any way this could be fixed.

This also potentially means maintaining the LLVM bindings for two LLVM versions

We already maintain¹ support for LLVM versions 3.7 through 3.9 (and potentially trunk). As long as Xcode’s LLVM is not some ancient version, I think we’re good in that regard. If Xcode LLVM is really some ancient/custom/etc version, then I don’t see us being able to support this feature at all, then. Especially since we have no option of sending patches to that LLVM in order to add the features we need. I also wouldn’t want to lock rustc into supporting 3.7 forever in case Apple decided to not update Xcode LLVM until 2038.

¹: however, if rustc was built against LLVM x.y, it must be linked to LLVM x.y exactly.

[brson]

Dynamically loading LLVM is certainly not an option. The LLVM API is less stable than its bitcode and, if we build against LLVM x.y, then having LLVM x.z instead of x.y will almost certainly break rustc. Just link to the system LLVM statically (like --llvm-root already does).

@nagisa The C++ API is unstable, but we use the C API and have had a lot of success supporting multiple versions of LLVM at once. I don't see the difference in terms of API support.

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

We could just blanket ship Apple's LLVM for all apple platforms, but this means coupling our LLVM to Apples for even desktop machine code generation, and precludes the option of supporting iOS bitcode on non-apple hosts.

Okay… so, the thing is that LLVM bitcode format is kinda stable between versions of LLVM. Does apple do something unusual to make that not true, or is the bitcode format used not LLVM altogether (i.e. has apple specific stuff)?

The bitcode format is not stable between versions.

does distributing native binary libraries not work anymore for iOS?

It does work today. It is not the preferred method and it's not obvious it will continue to be supported.

This also potentially means maintaining the LLVM bindings for two LLVM versions (not necessarily two neighbouring minor versions, either). I don't think it's valid to just use an old version of apple-llvm.

@ricky26 We successfully maintain compatibility between several versions of LLVM. As long as Apple's and ours don't drift too far apart it should be doable, but there's always the risk of such great breakage that the divide can't be crossed, and I know there are major API changes coming.

[bluejekyll]

As long as Xcode’s LLVM is not some ancient version, I think we’re good in that regard.

From this page https://gist.github.com/yamaya/2924292:

clang-700.0.72 => LLVM 3.7.0
clang-700.1.76 => LLVM 3.7.0
clang-700.1.81 => LLVM 3.7.0
clang-703.0.29 => LLVM 3.8.0
clang-703.0.31 => LLVM 3.8.0

[nagisa]

The C++ API is unstable, but we use the C API and have had a lot of success supporting multiple versions of LLVM at once. I don't see the difference in terms of API support.

That is not true. We have (quite large!) a number of bindings to C++ APIs in form of rustllvm. There’s a number of cases where we compile that wrapper depending on the version of LLVM compiled against. In case version of LLVM used and compiled against does not match, you’ll get dynamic linker errors, or worse, run into issues at runtime.

precludes the option of supporting iOS bitcode on non-apple hosts.

If Apple does not want to take bitcode generated by anything else than their fork of LLVM, then I don’t see how we could do anything here short of maintaining a similar fork and reverse-engineering their internal patches.

The bitcode format is not stable between versions.

Sure, but it is quite fair to assume¹ that bitcode between various revisions of LLVM known as 3.7.0, for example, is sufficiently compatible for purposes of generating bitcode for consumption by another build of LLVM from 3.7.0 series. Its certainly better than linking to libLLVM dynamically.

¹: especially given that bitcode from 3.2 series is still compatible with 3.8 LLVM, even if its a very small specimen.

[comex]

Some notes:

• Xcode doesn't even ship with a linkable libLLVM (static or dynamic), only a libclang.dylib which statically links to LLVM.
• Apple does ship the source to their 'clang' (including LLVM) under 'Developer Tools' on https://opensource.apple.com, but that site tends to take forever to be updated.
• The story may be better with Swift, which has its own fork of LLVM with release tags - but they may not correspond perfectly to what's shipped with Xcode. (It's possible to use open-source toolchain snapshots with Xcode, but projects built with such toolchains can't be submitted to the App Store.)

[aturon]

cc @rust-lang/compiler

[mitsuhiko]

Would be curious to hear how other programming languages plan on dealing with this. In particular mono and go.

[ricky26]

Unity's answer to this problem is il2cpp - building all of their IL assemblies into C++ code.

Relevant golang bug: golang/go#12682; the suggestion there seems to be that they could use the LLVM go toolchain (which isn't as featured as the standard go toolchain).

All in all, the story for bitcode support outside of Apple is poor.

[mitsuhiko]

Mono proper goes via apple LLVM it would appear: http://tirania.org/blog/archive/2015/Sep-02.html

[mitsuhiko]

One stumbling block will be that you cannot carry inline assembly in bitcode :(

For the mono story I had a quick exchange with Miguel de Icaza about what Mono does for the curious: https://twitter.com/mitsuhiko/status/769458873237434368

[comex]

@mitsuhiko You can have inline assembly in bitcode on iOS and tvOS, but not watchOS, for some reason.

[russellmcc]

Any movement on this? I don't feel at all good about using Rust on iOS without a plan for supporting bitcode. Apple has a history of making optional things like this non-optional fairly suddenly, and indeed bitcode is already required on watchOS and tvOS.

I feel a bit like all of the recommended solutions are somewhat icky. The least messy way I can think of is to allow different targets to overload the codegen behaviour and have the Apple codegen path be in a dynamic crate. (Which would just be the regular codegen path compiled against apple-llvm.)

This approach (by @ricky26) seems to be the most natural to me as a rustc user.

[alexcrichton]

I don't believe anything has changed on this recently to my knowledge at least. With LLVM's recent announcement about versioning they indicated that bitcode should (I think) always be loadable by future versions of LLVM. That may mean that this issue is "solved" on a fundamental level, but it'd still require a more ergonomic interface to extract all the bitcode.

[adrianbrink]

Are there any updates on this?

[samkhawase]

This commenter on HackerNews has succeeded in using Bitcode generated from Rust on macOS and iOS. The thread has some detailed information on how to enable bitcode for rust binaries, which sounds like a great news!

https://news.ycombinator.com/item?id=14305084

[comex]

As the commenter in question, quick notes:

• I used -C lto --emit llvm-bc to get rustc to emit one .bc file containing the current crate and all dependencies. This works, but is essentially a hack; in particular, it doesn't work for C dependencies, including jemalloc (though that isn't used on iOS anyway). It would be better if rustc properly supported emitting Mach-Os with "embedded bitcode"; you can look at the clang source to see how it's done.

• If you want to try it and don't mind the hack, it does seem to 'just work', except for the version issue:

• The main practical obstacle is that Rust syncs with LLVM trunk more frequently than Xcode, which seems to only do so yearly. Newer LLVM versions can load old bitcode, but not the other way around, so you have to either use an old version of rustc or build the latest against an old LLVM. (Actually, rustc 1.17 seems to still work with Xcode 8.x, but not nightly since the LLVM 4.0 upgrade; that's just coincidence though.)

• Ideally Rust would ship official binaries built against a suitable LLVM version. In practice, it seems to be fine to use the correct version of stock LLVM, but if you do want to use Apple's fork:

  • As I said in a previous comment, Xcode doesn't ship LLVM binaries in any linkable form, even dynamically; only clang and libclang.dylib (which exports the Clang API but not the LLVM one).
  • But, as I also said, the source code usually goes up on opensource.apple.com (under Developer Tools -> clang; the archive includes all of LLVM), just somewhat inconsistently/after a delay. However, especially given the expanded backwards compatibility guarantees noted by @alexcrichton, it may not be the end of the world to not always be up to date. Currently, the most recent source release is for Xcode 8.2.1, whereas the latest is 8.3.2.

[volodg]

Does bitcode work now? Did anybody try?

[amrox]

@volodg I'm a rust novice. However, I ran through this tutorial with the latest rust nightly (at time of writing rustc 1.37.0-nightly (088b98730 2019-07-03)) and it did NOT work.

It seems like the markers are there..

$ otool -arch arm64 -l librust.a  | grep bitcode
  sectname __bitcode
...

But I get the following error when building for iOS device (simulator works):

ld: '/Users/amrox/Documents/Projects/rust-ios-example/hello-rust/libs/librust.a(rust-e6011ffb55678675.rust.8yq9vjk7-cgu.3.rcgu.o)' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

That's as far as I got.

[saturday06]

I succeeded to link rustc nightly-2019-09-05 aarch64-apple-ios and rustflags = "-C lto -Z embed-bitcode" staticlib with clang-1100.0.33.5 (Xcode 11 beta 7) and -fembed-bitcode app. Source code is https://github.com/saturday06/rust-ios-bitcode-test .

[thombles]

• Do you know if the __cmdline section is necessary?

Two years later I can answer this - for deploying to physical devices or creating Xcode archives an empty cmdline is fine, however for App Store submissions Apple performs validation of the clang command line. I’ve put a longer description in this PR, which includes a hacky patch to make it work: getditto/rust-bitcode#7

I’d like to upstream this somehow but the options are not appealing. Do we make up some clang options when targeting iOS? Do we provide an envvar to choose exactly how to lie in case Apple changes their validation rules? I’d be happy to make the change if there’s a reasonable choice here.

[Narniab]

-Cembed-bitcode=no does not work for ios target. there are still bitcode section in .a file
how to build without bitcode?

[fzyzcjy]

Hi, after years, is there any updates? Thanks! In other words, is there anyone who successfully make an app with Rust to deploy to apple store?

[extrawurst]

We have an app using rust In the stores. No Bitcode required unless you want to support Apple Watch so far

[fzyzcjy]

@extrawurst Thank you very much! So if I understand correctly: I just need to disable bitcode, and everything would be fine?

[thombles]

If you do want the bitcode for some reason or another, the scripts we maintain at $work continue to function for building iOS apps with bitcode and submitting to the app store, up to Xcode 13. More broadly though I'm not aware of any changes regarding Apple's forks of LLVMs and the difficulty of making this work with mainline Rust development.

[fzyzcjy]

@thombles Thank you!

[saks]

I've encountered an issue that looks similar:

ld: Invalid record (Producer: 'LLVM13.0.0-rust-1.59.0-stable' Reader: 'LLVM APPLE_1_1300.0.29.30_0') for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I'm pretty sure that bitcode is disabled, since it works if I use rust 1.55 (before upgrade to LLVM 12) and xcode project is configured to disable bitcode.
I'd appreciate if anyone can suggest how to debug this further. Thanks!

[bernaferrari]

Bitcode is the future of Apple's app distribution

Not anymore 😛

https://developer.apple.com/documentation/xcode-release-notes/xcode-14-release-notes

Starting with Xcode 14, bitcode is no longer required for watchOS and tvOS applications, and the App Store no longer accepts bitcode submissions from Xcode 14.

Xcode no longer builds bitcode by default and generates a warning message if a project explicitly enables bitcode: “Building with bitcode is deprecated. Please update your project and/or target settings to disable bitcode.” The capability to build with bitcode will be removed in a future Xcode release. IPAs that contain bitcode will have the bitcode stripped before being submitted to the App Store. Debug symbols for past bitcode submissions remain available for download. (86118779)

[jzxchiang1]

https://developer.apple.com/documentation/xcode-release-notes/xcode-14-release-notes

Does that mean we no longer need to use a specialized toolchain like https://github.com/getditto/rust-bitcode, and can instead safely disable bitcode in our XCode app?

[bernaferrari]

Yes.

[workingjubilee]

@thombles makes a good point here:

It's exciting news but let's not count our chickens before they hatch. I'll be happy to make some sort of announcement after we've seen:

libraries produced by vanilla Rust toolchains working in Xcode 14 GM on physical devices
successful deployments all the way to TestFlight or the App Store, thereby exercising all of Apple's binary analysis checks

It should work just fine since the no-bitcode path has been reliable so far, but a lot can happen over the course of an Xcode beta.

I'd also take "Xcode 15 is announced", on the assumption that if anything goes awry we'll hear about it by then, but if anyone wants to reap this issue before then, they're welcome to it.

[jfro]

fwiw we currently ship macOS (non-store) & iOS (TestFlight & Store) with a Rust library driving most of our core functionality. We've never used bitcode since we didn't target any of the previously bitcode requiring platforms.
We'll be shipping an update in the near future that'll also be built with 14 I believe (think they're accepting 14 builds now...)

[thombles]

No surprises really, but I've confirmed that the standard Rust toolchain is working fine all the way to App Store validation using Xcode 14. I've added a notice to rust-bitcode to advise users it's no longer needed. Dare we... close this? :)

[nagisa]

Closing as per the comment above.