Hermes Engine takes 2.5 to 19 times longer to process promises than Chakra

[AlexLablaiksSAP]

Problem Description

We have recently switched to Hermes to leverage Direct Debugging via VSCode. Unfortunately, we have been noticing that large datasets have been taking around 20 times longer in Hermes to process promises verses Chakra. Specifically, datasets that take 1.2 seconds with Chakra verses 25 seconds with Hermes.

I have been able to reproduce this with a simple React Native app available at hermes-promise-test, which uses version 0.68.0. Our metadata application often processes fields based on various rule sets. I have replicated this in the aforementioned repository by formatting the data set field values to uppercase or lowercase. The results are as follows:

1. Flat (one promise per field) of 1,000 objects: 73.29 ms vs 4.71 ms or 15.5 times longer for Hermes.
2. Flat of 10,000 objects: 748 ms vs 23 ms or 32 times longer for Hermes.
3. Nested (each object has a Promise.all() for every field (6)) of 1,000 objects: 379 ms vs 9 ms or 41 times longer for Hermes.
4. Nested of 10,000 objects: 23626 ms vs 145 ms or 163 times longer for Hermes.

Steps To Reproduce

1. Checkout hermes-promise-test
2. Observe Hermes and Chakra differences by toggling between the two engines by pressing Ctrl + Shift + D and selecting "<Enable/Disable> Remote JS Debugging".
3. Choose Mock Data with Flat Promises or Mock Data with Nested Promises
4. Click either Get 1k Object Cells or Get 10k Object Cells to populate data.
5. Click the Format button to process the data set. Note that times will appear in the console log.

Expected Results

Hermes to perform similar or better than Chakra.

CLI version

7.0.3

Environment

System:
    OS: Windows 10 10.0.19042
    CPU: (16) x64 Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
    Memory: 16.96 GB / 31.75 GB
  Binaries:
    Node: 16.14.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.15 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 8.3.1 - C:\Program Files\nodejs\npm.CMD
    Watchman: Not Found
  SDKs:
    Android SDK:
      API Levels: 28, 29, 30
      Build Tools: 28.0.3, 29.0.2, 30.0.3, 31.0.0
      System Images: android-30 | Google APIs Intel x86 Atom
      Android NDK: Not Found
    Windows SDK:
      AllowDevelopmentWithoutDevLicense: Enabled
      AllowAllTrustedApps: Enabled
      Versions: 10.0.16299.0, 10.0.18362.0, 10.0.19041.0
  IDEs:
    Android Studio: Version  4.2.0.0 AI-202.7660.26.42.7351085
    Visual Studio: 16.11.32228.343 (Visual Studio Enterprise 2019)
  Languages:
    Java: 11.0.10 - C:\Program Files\Microsoft\jdk-11.0.10.9-hotspot\bin\javac.EXE
  npmPackages:
    @react-native-community/cli: Not Found
    react: 17.0.2 => 17.0.2
    react-native: 0.68.0 => 0.68.0
    react-native-windows: 0.68.0 => 0.68.0
  npmGlobalPackages:
    *react-native*: Not Found

Target Platform Version

10.0.19041

Target Device(s)

Desktop

Visual Studio Version

Visual Studio 2019

Build Configuration

Debug

Snack, code example, screenshot, or link to a repository

https://github.com/AlexLablaiksSAP/hermes-promise-test

[AlexLablaiksSAP]

hermes-promise-test.xlsx

[NickGerleman]

From your steps, you mentioned opening the RNW developer menu. Are you running a debug build of your application?

Chakra exposes it's APIs to react-native-windows in the form of ABI-safe C functions. It lets RNW debug builds use a release version of Chakra. The Hermes boundary is not yet ABI safe, and MSVC has different debug snd release ABIs. This currently forces RNW to use an "optimized debug build" of Hermes, which performs worse than its release optimized build.

This constraint go away I think, with work to move JS engine interface to be ABI safe. But right now Hermes direct debugging, like web debugging, is not representative of release build performance. Debug-time developer experience is still super important though. Did switching to Hermes direct debugging degrade that for your app?

[acoates-ms]

Also from your steps you are not comparing Chakra vs Hermes at all. When you enable remote debugging, you are running the JS inside a webbrowser (either Edge or Chrome probably) - both of which use V8. When running the JS in a browser you are running inside a very different JS environment including a completely different promise implementation. There are so many differences between how things run in "Remote Debugging" env, that it should not be used for any kind of performance testing.

[AlexLablaiksSAP]

From your steps, you mentioned opening the RNW developer menu. Are you running a debug build of your application?

Chakra exposes it's APIs to react-native-windows in the form of ABI-safe C functions. It lets RNW debug builds use a release version of Chakra. The Hermes boundary is not yet ABI safe, and MSVC has different debug snd release ABIs. This currently forces RNW to use an "optimized debug build" of Hermes, which performs worse than its release optimized build.

This constraint go away I think, with work to move JS engine interface to be ABI safe. But right now Hermes direct debugging, like web debugging, is not representative of release build performance. Debug-time developer experience is still super important though. Did switching to Hermes direct debugging degrade that for your app?

Yes, all tests were performed with a Debug build initially and were essentially live switched using the developer menu.

For our actual application, which my test repository is imitating, we do indeed have about a 20 second load screen time which is similar to the 10K Nested Promise test in the referenced repository.

I have gone back and rebuilt and tested in a Release build. The performance of Hermes in Release was still worse than webdebugging. It's still about 8 seconds for a nested 10K promise test. The others are under 1 second but still have the worst performance. I have updated the application to alert with the measure and have updated the spreadsheet with the data.
hermes-promise-test.xlsx

[AlexLablaiksSAP]

Also from your steps you are not comparing Chakra vs Hermes at all. When you enable remote debugging, you are running the JS inside a webbrowser (either Edge or Chrome probably) - both of which use V8. When running the JS in a browser you are running inside a very different JS environment including a completely different promise implementation. There are so many differences between how things run in "Remote Debugging" env, that it should not be used for any kind of performance testing.

I can go ahead and provide you with updated results from Chakra in release for the repository in question.

[AlexLablaiksSAP]

I have tested Chakra with both Debug and Release. Results are as follows:
For Debug, Hermes takes 3.5 to 19 times longer than Chakra.
For Release, Hermes takes 2.5 to 7.7 times longer than Chakra.

Important to note; for the larger Nested Promise tests, Chakra Debug outperforms Hermes in Release.
hermes-promise-test.xlsx

[vmoroz]

@AlexLablaiksSAP I am reopening the issue.
As we discussed with your team today, I clearly see the same issue playing with your repro.
From what I see in the WPA (Windows Perf Analysis tooling), where are no significant delays in the JS thread. The most of time is spent inside of the Hermes interpreter. The function that seems to be the most expensive is the Array.indexOf().

The next steps:

• Discuss this issue with Meta's Hermes team
• I would like to see the Hermes CPU sampling profiler results. I need to use the latest Hermes bit for it.