[v0.75.2] macOS (Catalyst) builds fail with hermes.framework: bundle format is ambiguous (could be app or framework)

[birdofpreyru]

Description

Migrating my libraries / app from RN@0.74 to the latest RN@0.75.2. When building for macOS (Catalyst) target my build fails with that error hermes.framework: bundle format is ambiguous (could be app or framework). I first posted the issue to the Hermes repo (facebook/hermes#1492), but I was told there, it must be an issue on RN side, how it packages stuff.

Steps to reproduce

N/A

React Native Version

0.75.2

Affected Platforms

Build - MacOS

Output of npx react-native info

N/A

Stacktrace or Logs

N/A

Reproducer

N/A

Screenshots and Videos

No response

[react-native-bot]

⚠️	Missing Reproducible Example
ℹ️	We could not detect a reproducible example in your issue report. Please provide either: If your bug is UI related: a Snack If your bug is build/update related: use our Reproducer Template. A reproducer needs to be in a GitHub repository under your username.

[react-native-bot]

⚠️	Missing Reproducible Example
ℹ️	We could not detect a reproducible example in your issue report. Please provide either: If your bug is UI related: a Snack If your bug is build/update related: use our Reproducer Template

[birdofpreyru]

@sunshinezxq that looks likes an attempt to trick users into downloading a malware, I got it reported to GitHub :)

[cipolleschi]

Thanks for reporting the user.

QQ: How do I repro it?
I create a new app and try to archive for macCatalyst?

[birdofpreyru]

QQ: How do I repro it?
I create a new app and try to archive for macCatalyst?

Yes, I just verified, a clean app created with @react-native-community/cli, macCatalyst build fails exactly this same way for me.

[zahidhussain998]

Yeah same thing is happening here

[cipolleschi]

I just created a new app with 0.75.2 and I can't reproduce the issue for catalyst (See the screenshot).

We had a similar report back then, but I couldn't repro either.

Can you try run pod deintegrate and try again. I'm afraid that cocoapods cached a wrong version of hermes in your system and tries to use that one to run macCatalyst for some reason.

[birdofpreyru]

@cipolleschi

pod deintegrate did not help.

I tried to remove all caches I may think of — all installed / generated files in the clean app template, entire ~/Library/Caches/CocoaPods, and ~/Library/Developer/Xcode/DerivedData folders — it did not help either.

Have you tested it with the new architecture build? May it be caused by me still having an old Intel-based mac mini, rather than latest Apple Silicon one?

[cipolleschi]

I tried both New Arch and Old Arch and it works. Although it should not make a difference for Hermes which is architecture agnostic.

The intel-based mac mini could be an option, I have not thought about the possibility, honestly. 🤔
Also, I'm not sure how I could test that hypothesis... What is strange is that we didn't really changed anything in the Hermes build logic that could make that fail. And the catalyst architecture we build for CatalystMacOS is arm64_x86_64, so it should support both silicon (arm64) and intel (x86_64)...

[birdofpreyru]

@cipolleschi Here is the detailed error I get:

and here is the archived content of that hermes.framework folder within the bundled app:
hermes.framework.zip
if you see anything unexpected about these?

Anything else I can check?

[cipolleschi]

Interestingly it might be a symlink issue: https://stackoverflow.com/questions/25969946/osx-10-9-5-code-signing-v2-signing-a-framework-with-bundle-format-is-ambiguou

[cipolleschi]

the Catalyst slice of a xcframework has a structure that is similar to the one used by a desktop app.

The various hermes files with the empty doc icon are symlink to executable. I think your system does not interpret them properly and see them just as files with a link inside... 🤔

I need to spend more time looking at how we move stuff around to see if, perhaps we can solve this at build time.

[birdofpreyru]

well, the Anatomy of Framework Bundle, referenced in the StackOverflow thread you referred to, says that Resources folder on top level of a proper framework bundle should be a symlink, and Versions should be the only real directory... that's not what I end up with - in my case Resources is just a regular folder with Info.plist file inside it. Not sure, if/how I can manually force it to be a symlink, to double-check it is the problem, though.

[cipolleschi]

in my case it is a folder as well, but it is working fine. could you confirm that the folder structure is the same?

[birdofpreyru]

The only difference I see is that hermes file in the root has 7MB size in my bundle. The ones under Versions/X have 13.6MB, the same as on your screenshot.

[cipolleschi]

The only difference I see is that hermes file in the root has 7MB size in my bundle

That's weird. They should all have the same size.

When you run pod install, do you have this line?

Also, are you building for Debug or Release?

[birdofpreyru]

Yes, I have that line:

I am trying it all with Debug build.

Looking into .tgz archive it pulls, I see there are different versions of hermes binary for different architecture variants, some have 13 MB, but there are several, e.g. under universal/hermes.framework/ios-arm64/hermes.framework, that have 6.5 MB size. I guess something caused my Xcode to pick and try to bundle a wrong one? I don't know much about toolchains, but I guess it evaluates on my machine it needs a toolchain that does not match any of the intended one, then it attempts to fallback to ios-arm64 or xros-arm64, whatever these mean, and then fails?

[birdofpreyru]

I looked more into my logs, I believe it copies the correct 13.6 MB into the app bundle (I found it says Selected xcframework slice ios-arm64_x86_64-maccatalyst), I guess it shrinks it down to ~7MB when it does this:

[birdofpreyru]

I guess, that stripping step does not take place if you target new Apple Silicon hardware? Btw, although the build fails, it turns out the app bundle it creates before the failure, runs seemingly fine 0.o

[cipolleschi]

I guess, that stripping step does not take place if you target new Apple Silicon hardware? Btw, although the build fails, it turns out the app bundle it creates before the failure, runs seemingly fine 0.o

Wut! if that's the case, it feels like an Xcode issue more than a RN one. Do you have any chance to test on a Silicon machine?

[birdofpreyru]

Do you have any chance to test on a Silicon machine?

Not yet; but I guess if I am starting to run into such problems, it is a high time to get myself a new Apple machine, which I'll probably do :)

[robertying]

The same error happens on my Apple Silicon mac as well, both for debug and archive.

[robertying]

Something definitely has changed from 0.74 to 0.75. These are the different hermes.tar.gz file structures after unzipping:

ls -al ~/Downloads/0.74.5/Library/Frameworks/universal/hermes.xcframework/ios-arm64_x86_64-maccatalyst/hermes.framework:

drwxr-xr-x@ 5 rying  staff  160 Aug  5 06:55 .
drwxr-xr-x@ 3 rying  staff   96 Aug  5 06:55 ..
lrwxr-xr-x@ 1 rying  staff   26 Aug  5 06:55 Resources -> Versions/Current/Resources
drwxr-xr-x@ 4 rying  staff  128 Aug  5 06:55 Versions
lrwxr-xr-x@ 1 rying  staff   23 Aug  5 06:55 hermes -> Versions/Current/hermes

ls -al ~/Downloads/0.75.2/Library/Frameworks/universal/hermes.xcframework/ios-arm64_x86_64-maccatalyst/hermes.framework:

drwxr-xr-x@ 5 rying  staff       160 Aug 20 02:49 .
drwxr-xr-x@ 3 rying  staff        96 Aug 20 02:49 ..
drwxr-xr-x@ 3 rying  staff        96 Aug 20 02:49 Resources
drwxr-xr-x@ 4 rying  staff       128 Aug 20 02:49 Versions
-rw-r--r--@ 1 rying  staff  13564883 Aug 20 02:49 hermes

[robertying]

Ok I think the symlinks are broken in the 0.75 hermes build.

To fix the symlinks, go to your project root and run:

cd ios/Pods/hermes-engine/destroot/Library/Frameworks/universal/hermes.xcframework/ios-arm64_x86_64-maccatalyst/hermes.framework
rm -r hermes Resources
ln -s Versions/Current/Resources Resources
ln -s Versions/Current/hermes hermes
cd Versions && rm -r Current && ln -s 0 Current # this is important

Verify hermes.framework looks like this:

drwxr-xr-x@ 5 rying  staff  160 Aug  5 06:55 .
drwxr-xr-x@ 3 rying  staff   96 Aug  5 06:55 ..
lrwxr-xr-x@ 1 rying  staff   26 Aug  5 06:55 Resources -> Versions/Current/Resources
drwxr-xr-x@ 4 rying  staff  128 Aug  5 06:55 Versions
lrwxr-xr-x@ 1 rying  staff   23 Aug  5 06:55 hermes -> Versions/Current/hermes

@cipolleschi does RN do the packaging for hermes or is this indeed a hermes issue (that we should forward back to the hermes repo)?

[hsjoberg]

does RN do the packaging for hermes or is this indeed a hermes issue (that we should forward back to the hermes repo)?

@robertying I believe OP was referred to the react-native github project by Hermes devs due to them not doing the packaging (facebook/hermes#1492).

FWIW, I'm seeing the same symptoms, but for react-native-macos. I'll try the symlink fixes soon.

[cipolleschi]

@robertying Thanks for your investigation. That helps a lot.

React Native is responsible to package and distribute Hermes, so that's actually a React Native issue.

What we changes w.r.t. 0.74 is that we had to move the whole build and release pipeline from CircleCI to GitHub Actions.
The upload artifact action has some issues with symlinks, sadly and it has given us headaches already in main as well.

I'll try to double check and do some extra checks today, but with React Universe coming up I'm not sure whether a fix will arrive soon. It might have to wait for a week or so.

Meanwhile, I'll reopen the issue.

[robertying]

Thanks @cipolleschi! Thankfully we have a workaround so it's not too urgent I guess.

[cipolleschi]

For (my) future references, Nightlies has the same issue. 🤔

➜  Nightly076 git:(main) ✗ ls -al /Users/cipolleschi/Tests/Nightly076/ios/Pods/hermes-engine/destroot/Library/Frameworks/universal/hermes.xcframework/ios-arm64_x86_64-maccatalyst/hermes.framework
total 26512
drwxr-xr-x  5 cipolleschi  staff       160 Aug 27 12:21 .
drwxr-xr-x  3 cipolleschi  staff        96 Aug 27 12:21 ..
drwxr-xr-x  3 cipolleschi  staff        96 Aug 27 12:21 Resources
drwxr-xr-x  4 cipolleschi  staff       128 Aug 27 12:21 Versions
-rw-r--r--  1 cipolleschi  staff  13573923 Aug 27 12:21 hermes

[cipolleschi]

Looking in the upload-artifacts repo, there are quite a few issues related to symlinks. It looks like that the default behavior for upload-artifacts is to follow symlinks and copy the file/folder the symlink points to. 🤦

I guess we have to add a custom step to revert those changes before creating the archive.

@hsjoberg that's expected, as the framework format for Catalyst and MacOS is the same

[cipolleschi]

#46301 this seems to fix the issue. It is not done yet, I only verified that there are symlinks in the generated artifacts.

[cipolleschi]

Hello there! I think I fixed the issue in the above PR.
Can some of you (@birdofpreyru @hsjoberg) please try it? 🙏

You can download the hermes tarball from this link

And then you can install pods using this command:

HERMES_ENGINE_TARBALL_PATH=<path-to-downloaded-tar> bundle exec pod install

and try to build again.

[gabrieldonadel]

Hello there! I think I fixed the issue in the above PR. Can some of you (@birdofpreyru @hsjoberg) please try it? 🙏

You can download the hermes tarball from this link

And then you can install pods using this command:

HERMES_ENGINE_TARBALL_PATH=<path-to-downloaded-tar> bundle exec pod install

and try to build again.

Hi @cipolleschi, not sure if we can use this link directly

But after pointing it to a local path I got pod install work, but my build would still fail, this time with a different error

[cipolleschi]

nono, you have to download the tarball locally first! 😄

[hsjoberg]

You can download the hermes tarball from this link

And then you can install pods using this command:

HERMES_ENGINE_TARBALL_PATH=<path-to-downloaded-tar> bundle exec pod install

and try to build again.

Thank you for looking into this issue @cipolleschi. 🙏
Is this the correct tarball though? It doesn't seem to contain the any xcframework, rather it's the actual hermes macOS desktop tools:

❯ unzip -vl hermes-osx-bin-Release.zip
Archive:  hermes-osx-bin-Release.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
10424467  Defl:N  2944140  72% 09-02-2024 18:31 ce126323  hermes
     479  Defl:N      311  35% 09-02-2024 18:31 0f360c6f  hermes-lit
 6175108  Defl:N  1700213  73% 09-02-2024 18:31 1e5631b3  hermesc
--------          -------  ---                            -------
16600054          4644664  72%                            3 files

Hermes Pod folder becomes empty after running HERMES_ENGINE_TARBALL_PATH=<path-to-downloaded-tar> bundle exec pod install:

TurboLnd/example/macos on  master [!?] took 15s
❯ HERMES_ENGINE_TARBALL_PATH=/Users/coco/Projects/Blixt/1/TurboLnd/example/macos/hermes-osx-bin-Release.zip bundle exec pod install
[...]
Installing hermes-engine (0.75.2)
[...]
Pod installation complete! There are 66 dependencies from the Podfile and 64 total pods installed.
TurboLnd/example/macos on  master [!?] took 15s
❯ ls Pods/hermes-engine

TurboLnd/example/macos on  master [!?]
❯

EDIT: I'm checking if I can find the correct file in https://github.com/facebook/react-native/actions/runs/10671527175/

EDIT2: I found this job with this artifact: https://github.com/facebook/react-native/actions/runs/10671527175/artifacts/1883232802 (slice-catalyst-Release.zip). Unfortunately I'm seeing the same error with that one, and the symlinks broken. Which job is the one I should try?

[cipolleschi]

Sorry, my bad. This is the right link

[gabrieldonadel]

Sorry, my bad. This is the right link

Even when using this new link I still get ReactCommon/hermes/inspector-modern/chrome/Registration.h:15:10: 'hermes/hermes.h' file not found

[hsjoberg]

@cipolleschi I'm not sure exactly what I should make of this, but it appears to almost work for react-native-macos.
I'm getting some undefined symbols, but perhaps react-native(-macos) 0.76 would be needed for it to work?

---

@gabrieldonadel did you unzip the zip-file first in order to get the tarball? Looks like it's failing pretty early.

[cipolleschi]

Can you guys try with a nightly? I already applied the same changes to main, so yesterday nightly contains all the proper fixes. If you can safely build catalyst or MacOS using the Nightly, we should be good to go.

@hsjoberg yes, MacOS might have to catch up on the changes my colleagues made for the Debugger. The inspector_modern symbol you see there is the inspector for the new Debugger in the React Native Dev Tools

[hsjoberg]

@hsjoberg yes, MacOS might have to catch up on the changes my colleagues made for the Debugger. The inspector_modern symbol you see there is the inspector for the new Debugger in the React Native Dev Tools

@cipolleschi Gotcha, yes that makes sense.
I think react-native-macos usually pulls in commits after there's a new release line for react-native core.

Pinging @Saadnajmi so that you're aware of this issue.

[Saadnajmi]

I don't expect React Native macOS to hit the same issue, since it's technically a different platform. But yes, working on merging up to 0.76 right now..

[hsjoberg]

Well, I'm currently seeing the same symptoms as OP on react-native-macos 0.75.

From what I can tell react-native-macos pulls in iOS Hermes, so I thought this was related to OPs issue:
https://github.com/microsoft/react-native-macos/blob/042cd6c36b23c21d20de628bc9756793e01ce282/packages/react-native/sdks/hermes-engine/hermes-utils.rb#L237-L257