fix: cjs-module-lexer WebAssembly out of memory fallback #43612

guybedford · 2022-06-28T23:00:47Z

This implements a possible fix for the Wasm memory issues that have been coming up with hosts that are unable to run WebAssembly in v8 due to low memory requirements.

See the thread here - nodejs/cjs-module-lexer#61, there have also been issues posted in core, some open and some closed.

I don't know why this took me so long, but we can just fallback to the JS lexer implementation if Wasm initialization fails, which this PR should provide.

//cc @nodejs/modules

nodejs-github-bot · 2022-06-28T23:00:51Z

Review requested:

@nodejs/modules

lib/internal/modules/esm/translators.js

ckcr4lyf · 2022-06-29T02:38:02Z

Seems it would fix the issue I was havving: #42222

Thanks a lot @guybedford

nodejs-github-bot · 2022-06-29T05:46:49Z

CI: https://ci.nodejs.org/job/node-test-pull-request/44945/

GeoffreyBooth · 2022-06-29T05:58:51Z

Is there some way to test this? Like some way to test that the JS version of the lexer returns the same results for some set of test cases? And/or test that the JS one actually gets loaded if the wasm init fails?

guybedford · 2022-06-29T06:00:42Z

Is there some way to test this? Like some way to test that the JS version of the lexer returns the same results for some set of test cases? And/or test that the JS one actually gets loaded if the wasm init fails?

The lexer project itself runs test cases against both versions, and we've been using both versions in production for a long time now.

In terms of simulating the Wasm memory error I wouldn't know to do that in a safe way on CI.

GeoffreyBooth · 2022-06-29T06:25:55Z

In terms of simulating the Wasm memory error I wouldn't know to do that in a safe way on CI.

You could do it via some external property, like if an environment variable USE_JS_CJS_LEXER is set then skip the wasm one; but it's not worth cluttering up production code.

nodejs-github-bot · 2022-06-29T07:30:48Z

CI: https://ci.nodejs.org/job/node-test-pull-request/44950/

nodejs-github-bot · 2022-07-03T17:33:56Z

Landed in 10ed3e8

PR-URL: #43612 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Minwoo Jung <nodecorelab@gmail.com>

maggie44 · 2022-09-06T08:58:39Z

This implements a possible fix for the Wasm memory issues that have been coming up with hosts that are unable to run WebAssembly in v8 due to low memory requirements.

See the thread here - nodejs/cjs-module-lexer#61, there have also been issues posted in core, some open and some closed.

I don't know why this took me so long, but we can just fallback to the JS lexer implementation if Wasm initialization fails, which this PR should provide.

//cc @nodejs/modules

@guybedford, should this fix apply to Docker container node:18.8.0-alpine3.16 on armv7hf builds? If so, it appears the issue still exists. Oddly, it isn't a problem using the exact same deployment on amd64 or arm64 containers to the same service.

failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory

➤ YN0001: │ RangeError: @babel/helper-hoist-variables@npm:7.18.6: WebAssembly.Instance(): Out of memory: wasm memory
at Sh (/build-context/.yarn/releases/yarn-3.2.2.cjs:196:231902)
at vh (/build-context/.yarn/releases/yarn-3.2.2.cjs:196:232663)
at /build-context/.yarn/releases/yarn-3.2.2.cjs:198:48232
at D5 (/build-context/.yarn/releases/yarn-3.2.2.cjs:392:17260)
at fn (/build-context/.yarn/releases/yarn-3.2.2.cjs:392:17295)
at Object.iNe (/build-context/.yarn/releases/yarn-3.2.2.cjs:423:4258)
at async Qo.fetchFromNetwork (/build-context/.yarn/releases/yarn-3.2.2.cjs:710:25275)
at async p (/build-context/.yarn/releases/yarn-3.2.2.cjs:429:1414)
at async m (/build-context/.yarn/releases/yarn-3.2.2.cjs:429:1769)
at async y (/build-context/.yarn/releases/yarn-3.2.2.cjs:429:2778)

guybedford · 2022-09-06T09:58:02Z

@Maggie0002 it looks like you're getting Yarn internals in the stack there so perhaps Yarn is running some Wasm? So I'm not sure this is the lexer causing the issue. If it were the lexer, then you would have a lexer line from the internal Node.js stack trace.

maggie44 · 2022-09-06T11:36:41Z

@Maggie0002 it looks like you're getting Yarn internals in the stack there so perhaps Yarn is running some Wasm? So I'm not sure this is the lexer causing the issue. If it were the lexer, then you would have a lexer line from the internal Node.js stack trace.

@guybedford, it's across a whole bunch. You think it could be a Yarn thing?

  failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
  
  ➤ YN0001: │ RangeError: @babel/compat-data@npm:7.18.13: WebAssembly.Instance(): Out of memory: wasm memory
      at Sh (/build-context/.yarn/releases/yarn-3.2.2.cjs:196:231902)
      at vh (/build-context/.yarn/releases/yarn-3.2.2.cjs:196:232663)
      at /build-context/.yarn/releases/yarn-3.2.2.cjs:198:48232
      at D5 (/build-context/.yarn/releases/yarn-3.2.2.cjs:392:17260)
      at fn (/build-context/.yarn/releases/yarn-3.2.2.cjs:392:17295)
      at Object.iNe (/build-context/.yarn/releases/yarn-3.2.2.cjs:423:4258)
      at async Qo.fetchFromNetwork (/build-context/.yarn/releases/yarn-3.2.2.cjs:710:25275)
      at async p (/build-context/.yarn/releases/yarn-3.2.2.cjs:429:1414)
      at async m (/build-context/.yarn/releases/yarn-3.2.2.cjs:429:1769)
      at async y (/build-context/.yarn/releases/yarn-3.2.2.cjs:429:2778)
  failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
  
  failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
  
  failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
  
  failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
  
  failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
  
  failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
  
  failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
  
  failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
  
  failed to compile wasm module: RangeError: WebAssembly.Instance(): Out of memory: wasm memory
  
  ➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory
      at z ([worker eval]:1:282761)
      at [worker eval]:1:331323
      at e ([worker eval]:1:331340)
      at getLibzipSync ([worker eval]:1:339972)
      at getLibzipPromise ([worker eval]:1:340034)
      at MessagePort.<anonymous> ([worker eval]:1:464293)
      at [nodejs.internal.kHybridDispatch] (node:internal/event_target:731:20)
      at exports.emitMessage (node:internal/per_context/messageport:23:28)
  ➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory
      at z ([worker eval]:1:282761)
      at [worker eval]:1:331323
      at e ([worker eval]:1:331340)
      at getLibzipSync ([worker eval]:1:339972)
      at getLibzipPromise ([worker eval]:1:340034)
      at MessagePort.<anonymous> ([worker eval]:1:464293)
      at [nodejs.internal.kHybridDispatch] (node:internal/event_target:731:20)
      at exports.emitMessage (node:internal/per_context/messageport:23:28)
  ➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory
      at z ([worker eval]:1:282761)
      at [worker eval]:1:331323
      at e ([worker eval]:1:331340)
      at getLibzipSync ([worker eval]:1:339972)
      at getLibzipPromise ([worker eval]:1:340034)
      at MessagePort.<anonymous> ([worker eval]:1:464293)
      at [nodejs.internal.kHybridDispatch] (node:internal/event_target:731:20)
      at exports.emitMessage (node:internal/per_context/messageport:23:28)
  ➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory
      at z ([worker eval]:1:282761)
      at [worker eval]:1:331323
      at e ([worker eval]:1:331340)
      at getLibzipSync ([worker eval]:1:339972)
      at getLibzipPromise ([worker eval]:1:340034)
      at MessagePort.<anonymous> ([worker eval]:1:464293)
      at [nodejs.internal.kHybridDispatch] (node:internal/event_target:731:20)
      at exports.emitMessage (node:internal/per_context/messageport:23:28)
  ➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory
      at z ([worker eval]:1:282761)
      at [worker eval]:1:331323
      at e ([worker eval]:1:331340)
      at getLibzipSync ([worker eval]:1:339972)
      at getLibzipPromise ([worker eval]:1:340034)
      at MessagePort.<anonymous> ([worker eval]:1:464293)
      at [nodejs.internal.kHybridDispatch] (node:internal/event_target:731:20)
      at exports.emitMessage (node:internal/per_context/messageport:23:28)
  ➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory
      at z ([worker eval]:1:282761)
      at [worker eval]:1:331323
      at e ([worker eval]:1:331340)
      at getLibzipSync ([worker eval]:1:339972)
      at getLibzipPromise ([worker eval]:1:340034)
      at MessagePort.<anonymous> ([worker eval]:1:464293)
      at [nodejs.internal.kHybridDispatch] (node:internal/event_target:731:20)
      at exports.emitMessage (node:internal/per_context/messageport:23:28)
  ➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory
      at z ([worker eval]:1:282761)
      at [worker eval]:1:331323
      at e ([worker eval]:1:331340)
      at getLibzipSync ([worker eval]:1:339972)
      at getLibzipPromise ([worker eval]:1:340034)
      at MessagePort.<anonymous> ([worker eval]:1:464293)
      at [nodejs.internal.kHybridDispatch] (node:internal/event_target:731:20)
      at exports.emitMessage (node:internal/per_context/messageport:23:28)
  ➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory
      at z ([worker eval]:1:282761)
      at [worker eval]:1:331323
      at e ([worker eval]:1:331340)
      at getLibzipSync ([worker eval]:1:339972)
      at getLibzipPromise ([worker eval]:1:340034)
      at MessagePort.<anonymous> ([worker eval]:1:464293)
      at [nodejs.internal.kHybridDispatch] (node:internal/event_target:731:20)
      at exports.emitMessage (node:internal/per_context/messageport:23:28)
  ➤ YN0001: │ RangeError: WebAssembly.Instance(): Out of memory: wasm memory
      at z ([worker eval]:1:282761)
      at [worker eval]:1:331323
      at e ([worker eval]:1:331340)
      at getLibzipSync ([worker eval]:1:339972)
      at getLibzipPromise ([worker eval]:1:340034)
      at MessagePort.<anonymous> ([worker eval]:1:464293)
      at [nodejs.internal.kHybridDispatch] (node:internal/event_target:731:20)
      at exports.emitMessage (node:internal/per_context/messageport:23:28)


...

PR-URL: nodejs/node#43612 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Minwoo Jung <nodecorelab@gmail.com>

PR-URL: #43612 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Minwoo Jung <nodecorelab@gmail.com>

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [node](https://github.com/nodejs/node) | stage | patch | `14.21.1-bullseye-slim` -> `14.21.2-bullseye-slim` | --- ### Release Notes <details> <summary>nodejs/node</summary> ### [`v14.21.2`](https://github.com/nodejs/node/releases/tag/v14.21.2): 2022-12-13, Version 14.21.2 'Fermium' (LTS), @richardlau [Compare Source](nodejs/node@v14.21.1...v14.21.2) ##### Notable Changes ##### OpenSSL 1.1.1s This update is a bugfix release and does not address any security vulnerabilities. ##### Root certificates updated to NSS 3.85 Certificates added: - Autoridad de Certificacion Firmaprofesional CIF [`A626340`](nodejs/node@A62634068) - Certainly Root E1 - Certainly Root R1 - D-TRUST BR Root CA 1 2020 - D-TRUST EV Root CA 1 2020 - DigiCert TLS ECC P384 Root G5 - DigiCert TLS RSA4096 Root G5 - E-Tugra Global Root CA ECC v3 - E-Tugra Global Root CA RSA v3 - HiPKI Root CA - G1 - ISRG Root X2 - Security Communication ECC RootCA1 - Security Communication RootCA3 - Telia Root CA v2 - vTrus ECC Root CA - vTrus Root CA Certificates removed: - Cybertrust Global Root - DST Root CA X3 - GlobalSign Root CA - R2 - Hellenic Academic and Research Institutions RootCA 2011 ##### Time zone update to 2022f Time zone data has been updated to 2022f. This includes changes to Daylight Savings Time (DST) for Fiji and Mexico. For more information, see <https://mm.icann.org/pipermail/tz-announce/2022-October/000075.html>. ##### Commits - \[[`436a596e99`](nodejs/node@436a596e99)] - **crypto**: update root certificates (Luigi Pinca) [#45490](nodejs/node#45490) - \[[`4b422d34af`](nodejs/node@4b422d34af)] - **deps**: V8: cherry-pick [`d2db7fa`](nodejs/node@d2db7fa7f786) (Richard Lau) [#45785](nodejs/node#45785) - \[[`625f4bf3a9`](nodejs/node@625f4bf3a9)] - **deps**: update corepack to 0.15.1 (Node.js GitHub Bot) [#45331](nodejs/node#45331) - \[[`48a9810de8`](nodejs/node@48a9810de8)] - **deps**: update corepack to 0.15.0 (Node.js GitHub Bot) [#45235](nodejs/node#45235) - \[[`9f4e64b603`](nodejs/node@9f4e64b603)] - **deps**: update timezone to 2022f (Richard Lau) [#45521](nodejs/node#45521) - \[[`f297b6bd21`](nodejs/node@f297b6bd21)] - **deps**: update archs files for OpenSSL-1.1.1s (RafaelGSS) [#45272](nodejs/node#45272) - \[[`11629fef15`](nodejs/node@11629fef15)] - **deps**: upgrade openssl sources to 1.1.1s (RafaelGSS) [#45272](nodejs/node#45272) - \[[`c3a90c4b44`](nodejs/node@c3a90c4b44)] - **http2**: fix memory leak when nghttp2 hd threshold is reached (rogertyang) [#41502](nodejs/node#41502) - \[[`785dc3efee`](nodejs/node@785dc3efee)] - **module**: cjs-module-lexer WebAssembly fallback (Guy Bedford) [#43612](nodejs/node#43612) - \[[`2dbeb889f6`](nodejs/node@2dbeb889f6)] - **node-api**: handle no support for external buffers (Michael Dawson) [#45181](nodejs/node#45181) - \[[`5b2ea124f3`](nodejs/node@5b2ea124f3)] - **test**: add test to validate changelogs for releases (Richard Lau) [#45325](nodejs/node#45325) - \[[`f13f889956`](nodejs/node@f13f889956)] - **test**: add a test to ensure the correctness of timezone upgrades (Darshan Sen) [#45299](nodejs/node#45299) - \[[`5608e6fa72`](nodejs/node@5608e6fa72)] - **tools**: update certdata.txt (Luigi Pinca) [#45490](nodejs/node#45490) - \[[`d6f1d7107b`](nodejs/node@d6f1d7107b)] - **tools**: have test-asan use ubuntu-20.04 (Filip Skokan) [#45581](nodejs/node#45581) - \[[`370a00f737`](nodejs/node@370a00f737)] - **tools**: make license-builder.sh comply with shellcheck 0.8.0 (Rich Trott) [#41258](nodejs/node#41258) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.walbeck.it/mwalbeck/docker-jellyfin-livestream/pulls/210 Co-authored-by: renovate-bot <bot@walbeck.it> Co-committed-by: renovate-bot <bot@walbeck.it>

nodejs-github-bot added esm Issues and PRs related to the ECMAScript Modules implementation. needs-ci PRs that need a full CI run. labels Jun 28, 2022

guybedford mentioned this pull request Jun 28, 2022

node ES6 import throws this error. nodejs/cjs-module-lexer#61

Closed

guybedford force-pushed the lexer-wasm-fallback branch from 4de4158 to 700056e Compare June 28, 2022 23:02

GeoffreyBooth approved these changes Jun 28, 2022

View reviewed changes

module: cjs-module-lexer WebAssembly fallback

a522525

guybedford force-pushed the lexer-wasm-fallback branch from 700056e to a522525 Compare June 29, 2022 00:07

VoltrexKeyva reviewed Jun 29, 2022

View reviewed changes

lib/internal/modules/esm/translators.js Outdated Show resolved Hide resolved

lint fix

e82b07e

VoltrexKeyva approved these changes Jun 29, 2022

View reviewed changes

lpinca approved these changes Jun 29, 2022

View reviewed changes

richardlau approved these changes Jun 29, 2022

View reviewed changes

BridgeAR approved these changes Jun 29, 2022

View reviewed changes

cjihrig approved these changes Jun 29, 2022

View reviewed changes

JakobJingleheimer approved these changes Jun 29, 2022

View reviewed changes

targos added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jun 30, 2022

JungMinu approved these changes Jun 30, 2022

View reviewed changes

lpinca added the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 3, 2022

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 3, 2022

nodejs-github-bot merged commit 10ed3e8 into nodejs:main Jul 3, 2022

targos mentioned this pull request Jul 12, 2022

v18.6.0 release proposal #43789

Merged

ckcr4lyf mentioned this pull request Jul 14, 2022

Feature/esm export motdotla/dotenv#644

Closed

This was referenced Jul 26, 2022

Potential Wasm execution limits in some low-memory environments #40201

Closed

RangeError: WebAssembly.instantiate(): Out of memory: wasm memory cross-seed/cross-seed#178

Closed

richardlau added lts-watch-v14.x labels Jul 26, 2022

targos removed the lts-watch-v16.x label Jul 28, 2022

targos mentioned this pull request Aug 3, 2022

v16.17.0 release proposal #44098

Merged

richardlau added backported-to-v14.x and removed lts-watch-v14.x labels Nov 24, 2022

richardlau mentioned this pull request Dec 7, 2022

v14.21.2 proposal #45775

Merged

guybedford mentioned this pull request May 14, 2023

Add RISC-V Support nodejs/cjs-module-lexer#86

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: cjs-module-lexer WebAssembly out of memory fallback #43612

fix: cjs-module-lexer WebAssembly out of memory fallback #43612

guybedford commented Jun 28, 2022

nodejs-github-bot commented Jun 28, 2022

ckcr4lyf commented Jun 29, 2022

nodejs-github-bot commented Jun 29, 2022

GeoffreyBooth commented Jun 29, 2022

guybedford commented Jun 29, 2022

GeoffreyBooth commented Jun 29, 2022

nodejs-github-bot commented Jun 29, 2022

nodejs-github-bot commented Jul 3, 2022

maggie44 commented Sep 6, 2022

guybedford commented Sep 6, 2022

maggie44 commented Sep 6, 2022 •

edited

Loading

fix: cjs-module-lexer WebAssembly out of memory fallback #43612

fix: cjs-module-lexer WebAssembly out of memory fallback #43612

Conversation

guybedford commented Jun 28, 2022

nodejs-github-bot commented Jun 28, 2022

ckcr4lyf commented Jun 29, 2022

nodejs-github-bot commented Jun 29, 2022

GeoffreyBooth commented Jun 29, 2022

guybedford commented Jun 29, 2022

GeoffreyBooth commented Jun 29, 2022

nodejs-github-bot commented Jun 29, 2022

nodejs-github-bot commented Jul 3, 2022

maggie44 commented Sep 6, 2022

guybedford commented Sep 6, 2022

maggie44 commented Sep 6, 2022 • edited Loading

maggie44 commented Sep 6, 2022 •

edited

Loading