RN 0.72-rc.3 release builds crash on startup due to bytecode version mismatch

[MicroDroid]

Description

Release builds broken

React Native Version

0.72.0-rc.3

Output of npx react-native info

info Fetching system and libraries information...
System:
  OS: Linux 5.15 Arch Linux
  CPU: (16) x64 AMD Ryzen 7 5800H with Radeon Graphics
  Memory: 9.04 GB / 31.35 GB
  Shell:
    version: "5.9"
    path: /bin/zsh
Binaries:
  Node:
    version: 18.14.2
    path: ~/.nvm/versions/node/v18.14.2/bin/node
  Yarn:
    version: 1.22.19
    path: /usr/sbin/yarn
  npm:
    version: 9.5.0
    path: ~/.nvm/versions/node/v18.14.2/bin/npm
  Watchman:
    version: 20221016.020512.0
    path: /usr/sbin/watchman
SDKs:
  Android SDK:
    API Levels:
      - "31"
      - "33"
      - "33"
    Build Tools:
      - 30.0.2
      - 30.0.3
      - 31.0.0
      - 33.0.0
      - 33.0.2
    System Images:
      - android-33 | Google APIs Intel x86_64 Atom
    Android NDK: Not Found
IDEs:
  Android Studio: AI-222.4459.24.2221.9971841
Languages:
  Java:
    version: 11.0.17
    path: /usr/sbin/javac
  Ruby: Not Found
npmPackages:
  "@react-native-community/cli": Not Found
  react:
    installed: 18.2.0
    wanted: 18.2.0
  react-native:
    installed: 0.72.0-rc.3
    wanted: 0.72.0-rc.3
npmGlobalPackages:
  "*react-native*": Not Found
Android:
  hermesEnabled: true
  newArchEnabled: false
iOS:
  hermesEnabled: Not found
  newArchEnabled: Not found

Steps to reproduce

npx react-native@0.72.0-rc.3 init TestApp --version 0.72.0-rc.3
cd ./TestApp
yarn start
# new terminal
yarn android --mode release
# watch it crash on your phone...
# new terminal
adb logcat
# observe a log entry about bytecode verison mismatch

Snack, code example, screenshot, or link to a repository

Use the official demo template

[kelset]

wasn't able to repro on my machine:

does it work for you when not using release mode?

[MicroDroid]

@kelset yes you need to use the release mode, and I also use a physical device but presumably that shouldn't matter given the error.

[cortinico]

observe a log entry about bytecode verison mismatch

Can you share the crash please?

[MicroDroid]

@cortinico

05-15 15:07:39.031 12479 12505 E AndroidRuntime: FATAL EXCEPTION: mqt_js
05-15 15:07:39.031 12479 12505 E AndroidRuntime: Process: com.testapp, PID: 12479
05-15 15:07:39.031 12479 12505 E AndroidRuntime: com.facebook.jni.CppException: Compiling JS failed: Wrong bytecode version. Expected 94 but got 95 Buffer size 761500 starts with: c61fbc03c103191f5f0000009f3ef2b1
05-15 15:07:39.031 12479 12505 E AndroidRuntime:        at com.facebook.jni.NativeRunnable.run(Native Method)
05-15 15:07:39.031 12479 12505 E AndroidRuntime:        at android.os.Handler.handleCallback(Handler.java:938)
05-15 15:07:39.031 12479 12505 E AndroidRuntime:        at android.os.Handler.dispatchMessage(Handler.java:99)
05-15 15:07:39.031 12479 12505 E AndroidRuntime:        at com.facebook.react.bridge.queue.MessageQueueThreadHandler.dispatchMessage(MessageQueueThreadHandler.java:27)
05-15 15:07:39.031 12479 12505 E AndroidRuntime:        at android.os.Looper.loop(Looper.java:236)
05-15 15:07:39.031 12479 12505 E AndroidRuntime:        at com.facebook.react.bridge.queue.MessageQueueThreadImpl$4.run(MessageQueueThreadImpl.java:228)
05-15 15:07:39.031 12479 12505 E AndroidRuntime:        at java.lang.Thread.run(Thread.java:923)

[cortinico]

Thanks for that @MicroDroid we'll look into trying to reproduce this locally.
In the meanwhile:

1. Have you, by any reason, installed a version of Hermes globally? Specifically, could you share the output of npm list -g
2. Are you on any proxy like VPN or are you using a corp network that might intercept the HTTP call to Maven Central (i.e. you have your own Maven proxy in your organization)?

[MicroDroid]

@cortinico

1. npm list -g and yarn global list both show a few packages and none of them are related to React Native in any way
2. No, and no.

[MicroDroid]

By the way, I followed the steps I wrote above to reproduce the error when getting the logcat above for you, so yeah the repro steps are exact, at least on my machine.

[cortinico]

Could you try on RC2 and RC1 and report if the issue is still happening?
Also could you try also on 0.71?

I've just verified that RN Tester on RC3 works fine in Release mode, so I'm unsure what's causing this.

[MicroDroid]

Yes, I already tried all 0.72-* versions, it happens on all, but on 0.71* it works fine (that's what I am using right now).

[cortinico]

I have the suspect this is somehow related to building on Linux. What is happening is that you're using a hermesc which outputs bytecode version (95) that is "newer" than the one expected by the hermes-engine runtime (94). Not sure why you end up getting a newer version but this requires further investigation.

[MicroDroid]

@cortinico Oh actually this is WSL, but anyways, the compiler itself is actually of version 95, it's supposed to be 94?

[MicroDroid]

Tested both win64 and linux64 binaries, both are 95

[Kudo]

@cortinico i can confirm the hermesc on linux is bytecode version 95. i don't know how react-native nowadays publishing/generating hermesc, i can help to investigate further tomorrow.

$ yarn why react-native
yarn why v1.22.19
[1/4] Why do we have the module "react-native"...?
[2/4] Initialising dependency graph...
[3/4] Finding dependency...
[4/4] Calculating file sizes...
=> Found "react-native@0.72.0-rc.3"
info Has been hoisted to "react-native"
info This module exists because it's specified in "dependencies".
info Disk size without dependencies: "68.81MB"
info Disk size with unique dependencies: "123.58MB"
info Disk size with transitive dependencies: "192.34MB"
info Number of shared dependencies: 195
Done in 0.83s.

$ ./node_modules/react-native/sdks/hermesc/linux64-bin/hermesc -version
LLVM (http://llvm.org/):
  LLVH version 8.0.0svn
  Optimized build

Hermes JavaScript compiler.
  Hermes release version: 0.12.0
  HBC bytecode version: 95

  Features:
    Debugger
    Contiguous Heap
    Zip file input

for reference the hermesc -version on macos is as following, we can seen both hermes release version and hermes bytecode version are different:

$ ./node_modules/react-native/sdks/hermesc/osx-bin/hermesc -version
LLVM (http://llvm.org/):
  LLVH version 8.0.0svn
  Optimized build

Hermes JavaScript compiler.
  Hermes release version: for RN 0.72.0-rc.3
  HBC bytecode version: 94

  Features:
    Zip file input

[cortinico]

Thanks for the clarification @Kudo. I believe this is an instance of cache poisoning on CircleCI where we reused hermesc from the latest build from main, when we should be isolating caches for each RN version. Fix is incoming

[Kudo]

good catch! thanks @cortinico.

[MicroDroid]

Do you guys have an ETA for when you're making a release that includes this fix?

[cortinico]

Do you guys have an ETA for when you're making a release that includes this fix?

This will be included in the next RC sometime next week or the following one

[Kudo]

double checked the linux hermesc on 0.72.0-rc.5. other than the Hermes release version: 0.12.0. everything looks aligned. i think the hermes release version does not really matter though.

LLVM (http://llvm.org/):
  LLVH version 8.0.0svn
  Optimized build

Hermes JavaScript compiler.
  Hermes release version: 0.12.0
  HBC bytecode version: 94

  Features:
    Debugger
    Contiguous Heap
    Zip file input

[cortinico]

i think the hermes release version does not really matter though

Yup that's a minor bug in us not overriding the hermes version number while building for Linux, so that shouldn't be a major blocker

[Kudo]

Yup that's a minor bug in us not overriding the hermes version number while building for Linux, so that shouldn't be a major blocker

thanks for the confirmation 👍