Why has Universal x86_64/arm64e support been so difficult?

[awwright]

I got an M1 mac (I was trying to get an Intel and didn't realize the ARM versions started shipping already, but that's another story), and I was disappointed to find that Homebrew didn't work out-of-the-box.

If I run brew through arch -x86_64 it works, but only installs x86_64 binaries. I would expect it to install a Universal binary where possible. Apple promised building these cross-platform executables would be dead simple.

I know it's possible, I was pleasantly surprised to find that all of Apple's tools (/usr/bin/git, etc) are dual-compiled:

$ file /usr/bin/git
/usr/bin/git: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/bin/git (for architecture x86_64):	Mach-O 64-bit executable x86_64
/usr/bin/git (for architecture arm64e):	Mach-O 64-bit executable arm64e

Is it really that difficult to add Universal binary support for Homebrew?

[carlocab]

I’m not familiar with the details, but I believe offering universal binaries is very difficult.

From my understanding Homebrew used to provide universal 32/64-bit binaries, when Apple still supported 32-bit machines. Things broke a lot.

[Rylan12]

Short answer: yes. If it was easy, we would have just done it. Take a look at Homebrew/brew#7857 to see just a snippet of the huge amount of work that's been done.

The good news, we're just about ready to officially announce support. You can choose to start using Homebrew natively or wait a few more days for official support. No date yet but soon.

[awwright]

Any specifics? That's one of the issues I read through trying to figure out what the problems are. I get there's going to be some problems when porting a lot of different libraries to a new architecture, but I haven't seen talk about Universal binaries at all.

[Rylan12]

Homebrew doesn't cross-compile anything. I, personally, can't give you specifics (I wasn't involved in any of that decision making/work). As Carlo mentioned, much of that comes from a back experience with universal binaries in the past. Sorry, that's really all I can give you just because of my own lack of familiarity with the issue.

[carlocab]

You haven't seen talk of universal binaries because Homebrew has no intention to offer universal binaries.

[awwright]

Er... OK, but why? Shipping Universal binaries seems like logical solution for M1 support, I mean, that's what Apple made it for. See why I'm curious about this?

[carlocab]

but why?

See my answer above:

Things broke a lot.

I also don't really understand this:

Shipping Universal binaries seems like logical solution for M1 support

Why is it logical? The logical thing to do would be to ship binaries designed for the machine they're meant to run on. That's what Homebrew is doing.

[awwright]

@carlocab I'm curious as to a technical reason. Or at least a few examples, if it's a lot of little things.

Why is it logical?

Shipping Universal binaries is the solution Apple is pushing. It's obvious they've put a lot of investment into getting it to work. With my intermediate level of knowledge, it's the first solution I would try to deploy. I'm curious what the roadblock is, so to speak. It sounds like nobody's even tried it.

And it's not like shipping a Universal binary is without benefit. If I could run /usr/local/bin/fish in either mode the same way I can run /bin/bash, that would be a huge feature for me.

[gromgit]

It sounds like nobody's even tried it.

Homebrew has done universal binaries before. PPC (or was it 32/64-bit?) UB support was removed after v1.9.0, and while it may have been removed due to Apple themselves dropping their own support, I hear the task of supporting universal binaries left a bad taste in maintainers' mouths.

Also, there's the question of bloat:

# Catalina
$ sw_vers -productVersion
10.15.7
$ /usr/bin/git --version
git version 2.24.3 (Apple Git-128)
$ file /usr/bin/git
/usr/bin/git: Mach-O 64-bit executable x86_64
$ ls -l /usr/bin/git
-rwxr-xr-x 1 root wheel 31488 Sep 22 05:57 /usr/bin/git

# Big Sur
$ sw_vers -productVersion
11.1
$ /usr/bin/git --version
git version 2.24.3 (Apple Git-128)
$ file /usr/bin/git
/usr/bin/git: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/bin/git (for architecture x86_64):	Mach-O 64-bit executable x86_64
/usr/bin/git (for architecture arm64e):	Mach-O 64-bit executable arm64e
$ ls -l /usr/bin/git
-rwxr-xr-x 1 root wheel 137536 Jan  1  2020 /usr/bin/git

A 4x blowup for the same version of git. I don't know how much larger a UB boost keg would get, or qt, or python, but I suspect Intel Big Sur users would not be happy to pay an "ARM (and a leg) tax".

[watkit]

I think Apple is pushing that because for many software developers who are working on full apps, having one app binary that can be portable across both architectures is useful. It does, however, increase the file size of those binaries.
In the case of homebrew, you're essentially installing the tools/libraries/software you need on that machine, which only supports one architecture (ie arm64 on Apple M1 Macs), so there's little benefit to including the two architectures in the same binary, especially when one architecture will only function after translation to Rosetta.

And it's not like shipping a Universal binary is without benefit. If I could run /usr/local/bin/fish in either mode the same way I can run /bin/bash, that would be a huge feature for me.

What's the benefit you're talking about here, of having code for both architectures in one Universal binary? I'm not sure it's clear from the above.

[awwright]

@gromgit Bloat is a perfectly valid concern, but there's no reason you have to ship Universal binaries to x86_64 users. I'm just investigating that as an option.

@watkit Sub-processes execute differently depending on if I'm launching from an x86_64 shell or an arm64e shell. Case in point: Try running homebrew inside arch -x86_64 bash vs. arch -arm64e bash.

[SMillerDev]

Bloat is a perfectly valid concern, but there's no reason you have to ship Universal binaries to x86_64 users. I'm just investigating that as an option.

Why would we bloat the systems of only some of our users? Technologically it's much easier for homebrew to continue with each system getting their own binary that is made for that specific configuration.

[carlocab]

I'm curious as to a technical reason.

From my (admittedly limited) understanding, shipping a universal binary requires either:

• cross-compilation; or,
• native compilation on separate machines, then stitching the two binary packages together after the fact.

Both are fragile and tricky to get right. (Doing either also introduces unnecessary complexity in Homebrew's CI workflows.) It's doable for a piece of software that you designed yourself and understand intimately. But for around 5000 different software packages, most of which you don't even know how to use? You're going to need a team of engineers working overtime for a while. (You're probably going to have to pay them too. That's not cheap.)

It sounds like nobody's even tried it.

That's because it's not even worth trying. It's a lot of hard work and effort in exchange for dubious benefit. If I had an application that already contained code that was native for my system, why would I want to keep the pieces that either

• don't run at all; or,
• require a translation layer in order for me to run them? (Note: this translation imposes a performance penalty.)

I know the word "universal" sounds fancy and exciting. Universal binaries are intended to simplify the distribution of software because it takes away the need to reliably check the architecture of the target platform. Homebrew, however, already has existing methods in place for checking system architecture, and so has zero need to take advantage of this simplification of distribution, especially not with its attendant complications.

[kencu]

other package management systems for macOS do offer universal arm64/x86_64 installs.

[MikeMcQuaid]

Those other package management systems may be a better fit for the use case of you and others in this thread.

[dcow]

That's because it's not even worth trying. It's a lot of hard work and effort in exchange for dubious benefit. If I had an application that already contained code that was native for my system, why would I want to keep the pieces that either

• don't run at all; or,
• require a translation layer in order for me to run them? (Note: this translation imposes a performance penalty.)

I know the word "universal" sounds fancy and exciting. Universal binaries are intended to simplify the distribution of software because it takes away the need to reliably check the architecture of the target platform. Homebrew, however, already has existing methods in place for checking system architecture, and so has zero need to take advantage of this simplification of distribution, especially not with its attendant complications.

This misses the nuance of what Apple is providing with rosetta2 a bit. It's not a lazy hack deployed to forgo the need to check target architecture. It's a feature that allows a machine to execute code compiled for either architecture. Current macOS systems with rosetta2 installed actually execute both architectures. They don't just consume the fat binaries only to use the aarch64e slice. This is required because it will be awhile before all software has native support for aarch64e. So if your terminal emulator is running under rosetta2 then everything it executes will be intel as well. And that's actually a feature... imagine developing software that is portable and wanting to debug the x86 version. Rosetta lets you do that.

Of course I want all my binaries to run natively on Apple silicon. However, I don't see us getting to that world for more than a few years. Think of people who just bought an Intel Macbook. They're not going anywhere anytime soon. And they will need x86_64 software for the lifetime of their machine. Requiring people maintain multiple installs of brew in a way works counter to the path provided by rosetta2.

[dcow]

But this is kinda the point. Since brew can't control everything installed on any given machine, it makes sense for brew to be a little less opinionated and err on the side of flexibility so as to support the type of scenarios that will arise during the transition.

I totally understand the desire to not deliver to users fat binaries by default and keep things thin and clean. However, I don't understand the aversion to incorporating logic into brew that would help users assemble fat bottles as needed for scenarios that require it.

[SMillerDev]

If my years as a maintainer have taught me one thing it's that any flexibility requires more support. With 20 maintainers for 5000 formulae, support resources are not a commodity.

There is simply no good solution here, we tried fat binaries in the ppc Era and it did not work as easily as you're describing now. This is why we continued the "we'll make a binary specific to your system configuration" that we already had. If you want the option to create fat binaries, make a pull request. I am not sure it'll be accepted, but it'll definitely be considered.

[dcow]

we tried fat binaries in the ppc Era and it did not work as easily as you're describing now

To be fair, I suspect it was a different strategy than what I'm suggesting regarding having brew locally lipo binaries client side below.

If you want the option to create fat binaries, make a pull request.

I generally prefer to discuss whether the maintainers would be interested in a feature, hence this discussion, prior to spending the time building it. That is what I was hoping to accomplish by expanding this discussion.

Finally, regardless of whether we would ever decide to add the additional functionality, I believe brew is broken when running under rosetta since it raises an error rather than downloading and installing an arm package (since that's the host arch). How would you recommend I get eyes back on Homebrew/brew#10302? Open a new bug report with additional context?

[SMillerDev]

I believe brew is broken when running under rosetta since it raises an error rather than downloading and installing an arm package (since that's the host arch).

For all intends and purposes the host arch is x86_64 when running in Rosetta.

If you want people to look at a different solution I'd recommend creating a new issue which does meet all requirements.

[dcow]

For all intends and purposes the host arch is x86_64 when running in Rosetta.

Ideally, yes. However a process can also know if it's being translated vai a sysctl call Apple added for this type of scenario. In practice, one can execute e.g. an arm version of tree from a rosette-invoked instance of e.g. kitty and things just work fine. The complexities of needing to architecture match only come into play if a program that is installed is Intel and needs to have intel dependencies to function. So it's actually a subtly different but important distinction. I'll create a bug report to discuss further.

[dcow]

It's also worth noting: brew doesn't have to even bottle fat binaries in its CI. M1 installs of brew could, at package install time on the local system, download both bottles (if available and requested by the user) and then lipo them into a single universal binary.. a growler.. if you will. It would reduce the problem to a client side install configuration option rather than having to rejigger the entire CI system.

[SMillerDev]

New possibilities have come to light and so we're having a new/continued discussion about how best to adapt.

They haven't really, you would just like a different solution and all the maintainers here are trying to explain why that isn't going to happen (unless you write it for yourself)

[dcow]

We did not consider the Fat binaries solution though back then.

New possibilities have come to light and so we're having a new/continued discussion about how best to adapt.

They haven't really

I mean @iMichka, also maintainer, admitted it wasn't considered at the time.

you would just like a different solution and all the maintainers here are trying to explain why that isn't going to happen (unless you write it for yourself)

Honestly @SMillerDev you're starting to sound antagonistic. I hear you about contributing this feature if it's something that I think would be valuable and it's something I'm considering. Just because multiple people have brought up the discussion about why brew doesn't support fat binaries doesn't mean that me engaging here comes from a position of entitlement thinking that maintainers should just do whatever feature I want end of story. I value discussion and exploration of complicated issues. And the current setup is genuinely confusing to me as evidenced by the issue I'd been trying to get to the bottom of regarding kitty and brew. I haven't demanded anything or suggested anybody failed or miscalculated or given anybody grief. Please don't assume things about my motivations and intentions.

[MikeMcQuaid]

I mean @iMichka, also maintainer, admitted it wasn't considered at the time.

It was considered and discussed on our private Slack, it just wasn't debated publicly. I shot this down hard because I remember working on our existing universal binary support for PPC/x86 (and x86/x86_64) and it was a massive pain that rarely worked.

Brew is rather opinionated whilst rosetta is agnostic and flexible.

This is to be expected. We have a team of ~30 part-time, volunteer maintainers who are unpaid. Apple is one of the richest corporations in the world and those working on Rosetta 2 are (presumably) well compensated and doing so during their working hours.

[dcow]

Would homebrew’s plugin system be capable of supporting this type of functionality? Sounds like you could use a bit of help and also aren’t really interested in tackling fat binaries so me maintaining a plugin might be the right balance.

[MikeMcQuaid]

You could probably make individual formulae that support this using lipo, yeh.

[skylersaleh]

If there isn't an intent to ever support universal binaries, would it be possible to at least have a sanctioned path for installing arm64 packages (preferably in the same locations that arm64 homebrew would install them at) on an x86 host with an x86 install of homebrew?

The current setup makes it very difficult to setup a cross compiling x86_64 to ARM environment (for compiling your own universal binaries) if you were using homebrew for installing your library dependencies.

[skylersaleh]

We don't recommend using Homebrew for installing library dependencies for many reasons.

I didn't know that using homebrew to install library dependencies for development was not a sanctioned usage model. I guess I need to look for other tools then.

[MikeMcQuaid]

We build binaries for the latest version of macOS and a bunch of stuff has Homebrew paths baked into it. If that's fine with you: cool, but it's not for a lot of folks.

[skakraze5]

We build binaries for the latest version of macOS and a bunch of stuff has Homebrew paths baked into it. If that's fine with you: cool, but it's not for a lot of folks.

It definitely is not the easiest thing to work around, but it is doable. And usually its worth it cuz homebrew tends to be the most up-to-date package manager for the libraries and tools I'm interested in. The process we use is as follows:

• Figure out what all the dependencies are for the library(s) that you are interested in
• If any customizations are needed for a particular dependency, pull down the formula for that bottle and add your customizations or just modify the formula with brew edit
• brew install each dependency with the --build-bottle option
• brew bottle each dependency
• Do the same thing for the library(s) you're interested in.
• extract each of the bottles
• remove any folders that aren't needed for client consumption (i.e. share folder)
• modify the dylibs to use @rpath for the their install path (this gets around the homebrew baked paths)
• Now you can just link against those libs and update your header search paths to find the header files

So right now the complication we have is we'd like those libraries to be universal, or to at least have easy access to the ability to build the arm64 bottles, even if not on an arm64 machine. We're obviously already doing a bunch of work to make these libraries consumable outside of homebrew itself. So if we could build all the bottles we need, we can use lipo make our own universal binary.

You can't run arm64 code on x86 so this wouldn't work. Various e.g. post_install blocks require running arm64 code

I get that, but as you can see from my example above, when worrying just about building the libraries, which is possible on either architecture, you're not as concerned about it running on that machine. post_install blocks aren't run when using the --build_bottle option. So really we're just looking for maybe an additional flag used in conjunction with the --build_bottle that allows you to build for a different architecture type.

I know there are options in the docs for using the --build-from-source and/or the --bottle-arch=, but so far I haven't had any luck using those options with the outcome I was looking for. I'm open to suggestions there.

If you have any advice on how to achieve my goals, either with homebrew directly or with some other way, I'm open to suggestions there too.

This all my seems a little fringe case, but in so many ways homebrew is so close to doing the whole library dependencies thing really well. Seems a shame to not invest a bit more in making it easier/possible.

[jonchang]

You'll have to use lipo or some other piece of software that better suits your needs.

Homebrew (i.e. Homebrew/brew) is developed primarily to enable the Homebrew maintainers to package and distribute software in its package repository (i.e. Homebrew/core). This use case (bundling bottles built by Homebrew maintainers and redistributing them in a downstream app) is not supported by Homebrew. Doing things such as cross-compiling for M1 Macs or creating universal binaries is also not supported.

You are right that cross-compiling and creating universal binaries could be included in Homebrew. However, as other maintainers have pointed out repeatedly in this discussion and elsewhere, there is no desire for the Homebrew maintainers to add these features as we don't use them in Homebrew/core due to past bad experiences with it on PPC and 32-bit x86.

The transition to ARM is difficult enough without also having to support the myriad problems with cross-compiling and universal binaries. If you think it is easy to support creating universal binaries for 4,000+ formulae, you are welcome to fork Homebrew/brew and homebrew/core and show the maintainers how wrong they are.

Ultimately, you are asking Homebrew maintainers to do a huge amount of unpleasant work for you for free, to support an uncommon use-case that is not needed for Homebrew.

3. Prioritise Maintainers Over Users
It’s important to be user-focused but ultimately, as long as you follow 1 above, Homebrew’s minimum number of users will be the number of maintainers. However, if Homebrew has no maintainers it will quickly become useless to all users and the project will die. As a result, no user complaint, behaviour or need takes priority over the burnout of maintainers. If users do not like the direction of the project, the easiest way to influence it is to make significant, high-quality code contributions and become a maintainer.

[MikeMcQuaid]

• If any customizations are needed for a particular dependency, pull down the formula for that bottle and add your customizations or just modify the formula with brew edit
• modify the dylibs to use @rpath for the their install path (this gets around the homebrew baked paths)

At this point you're better just writing a wrapper around all of Homebrew to do this for you or find something like e.g. MacPorts that does it better.

post_install blocks aren't run when using the --build_bottle option.

This is because they are run when bottles are poured. Some packages just won't work if the post_install is never run.

If you have any advice on how to achieve my goals, either with homebrew directly or with some other way, I'm open to suggestions there too.

Build from source outside of Homebrew. Integrate your dependencies into your existing build system. This is what I've done at every employer where been distributing Mac applications to users.

[jonchang]

I'm going to lock this as we seem to be circling back to the same points over and over again. See my response in #412 (reply in thread) for why we won't be supporting universal binaries.