[js/web] optimize module export and deployment #20165

fs-eire · 2024-04-01T09:11:45Z

Description

This PR make numbers of optimizations to onnxruntime-web's module export and deployment.

See each section below for more details.

Preview

onnxruntime-web@1.19.0-esmtest.20240513-a16cd2bd21

~~onnxruntime-web@1.19.0-esmtest.20240430-c7edbcc63d~~

~~onnxruntime-web@1.18.0-esmtest.20240428-624c681c83~~

~~onnxruntime-web@1.18.0-esmtest.20240411-1abb64e894~~

Breaking changes

There is no code change required, but there are a few differences regarding code import, flags, bundler config and deployment steps.

Importing:

Import table is changed. See following for details.

Current import table:

Target Name	Path for "import" or "require"	WebGL	JSEP	wasm	Proxy	Training
`ort` (default)	`onnxruntime-web`	✔️	❌	✔️	✔️	❌
`ort.all`	`onnxruntime-web/experimental`	✔️	✔️	✔️	✔️	❌
`ort.node`	`onnxruntime-web`	❌	❌	✔️	❌	❌
`ort.training`	`onnxruntime-web/training`	❌	❌	✔️	✔️^[1]	✔️
`ort.wasm`	`onnxruntime-web/wasm`	❌	❌	✔️	✔️	❌
`ort.wasm-core`	`onnxruntime-web/wasm-core`	❌	❌	✔️	❌	❌
`ort.webgl`	`onnxruntime-web/webgl`	✔️	❌	❌	✔️^[2]	❌
`ort.webgpu`	`onnxruntime-web/webgpu`	❌	✔️	✔️	✔️	❌

[1] didn't test. may not actually work.
[2] not working. this is a mistake in build config.

Proposed update:

Target Name	Path for "import" or "require"	WebGL	JSEP	wasm	Proxy	Training
`ort` (default)	`onnxruntime-web`	✔️	❌	✔️	✔️	❌
`ort.all`	~~`onnxruntime-web/experimental`~~ `onnxruntime-web/all`	✔️	✔️	✔️	✔️	❌
`ort.node`	`onnxruntime-web`	❌	❌	✔️	❌	❌
`ort.training`	`onnxruntime-web/training`	❌	❌	✔️	✔️	✔️
`ort.wasm`	`onnxruntime-web/wasm`	❌	❌	✔️	✔️	❌
~~`ort.wasm-core`~~	~~`onnxruntime-web/wasm-core`~~	❌	❌	✔️	❌	❌
`ort.webgl`	`onnxruntime-web/webgl`	✔️	❌	❌	✔️ ❌	❌
`ort.webgpu`	`onnxruntime-web/webgpu`	❌	✔️	✔️	✔️	❌

Flags:

The following flags are deprecated:

env.wasm.simd (boolean): will be ignored. SIMD is always enabled in build.

The following flags changed their type:

env.wasm.wasmPaths: When using this flag as a string ( for the URL prefix ), nothing is changed. When using this flag as an object ( for per-file path override ), the type changed:

-  export interface Old_WasmFilePaths{
-    'ort-wasm.wasm'?: string;
-    'ort-wasm-threaded.wasm'?: string;
-    'ort-wasm-simd.wasm'?: string;
-    'ort-training-wasm-simd.wasm'?: string;
-    'ort-wasm-simd-threaded.wasm'?: string;
-  };
+  export interface New_WasmFilePaths {
+    /**
+     * Specify the override path for the main .wasm file.
+     *
+     * This path should be an absolute path.
+     *
+     * If not modified, the filename of the .wasm file is:
+     * - `ort-wasm-simd-threaded.wasm` for default build
+     * - `ort-wasm-simd-threaded.jsep.wasm` for JSEP build (with WebGPU and WebNN)
+     * - `ort-training-wasm-simd-threaded.wasm` for training build
+     */
+    wasm?: URL|string;
+    /**
+     * Specify the override path for the main .mjs file.
+     *
+     * This path should be an absolute path.
+     *
+     * If not modified, the filename of the .mjs file is:
+     * - `ort-wasm-simd-threaded.mjs` for default build
+     * - `ort-wasm-simd-threaded.jsep.mjs` for JSEP build (with WebGPU and WebNN)
+     * - `ort-training-wasm-simd-threaded.mjs` for training build
+     */
+    mjs?: URL|string;
+  }

Bundler compatibility:

Config changes are need for bundlers. See usage example in /js/web/test/e2e/ for Webpack, parcel and rollup.

Deployment:

if consuming from a CDN, there is no breaking change.
if consuming from a local server, need to copy all ort-*.wasm and ort-*.mjs files (totally 6 files) in the dist folder. (previously only need to copy ort-*.wasm files.)

Problems

There are a few problems with the current module export and deployment:

Script URL cannot be correctly inferred when imported as ESM.
Workers are forcefully encoded using Blob URL, which makes onnxruntime-web not working in CSP environment and Node.js, when using proxy or multi-threading feature.
Generated JS code (by Emscripten) is encoded using function.toString(), which is unstable and error-prone.
When running with a different Emscripten build, always need the build step. Making it difficult to swap artifacts in deveopment/debug.

Goals

Full ESM support

Support variances of ways to import. Including:

import from HTML's <script> tag (IIFE format, exporting to global variable ort)

<script src="https://example.com/cdn-path-to-onnxruntime-web/dist/ort.min.js"></script>

import from source code inside <script type="module"> tag (ESM)

<script type="module">
  import * as ort from "https://example.com/cdn-path-to-onnxruntime-web/dist/ort.min.mjs";

  // using 'ort'
</script>

import in a CommonJS project (CJS format, resolve from package.json "exports" field)
```
// myProject/main.js
const ort = require('onnxruntime-web');
```

import in an ESM project (ESM format, resolve from package.json "exports" field)

// myProject/main.js (or main.mjs)
import * as ort from 'onnxruntime-web';

Support popular bundlers when importing onnxruntime-web into a CJS/ESM project.
- webpack (esm requires extra post-process step)
- rollup
- parcel (esm requires extra post-process step)
- More bundlers TBD
Multi-threading support for Node.js

NOTE: keeping single JavaScript file (the all-in-one bundle) is no longer a goal. This is because technically there is a conflict with the other requirements.

Important Design Decisions

Drop support of single JavaScript output.
- The current onnxruntime-web distribution uses a single JavaScript file to include all code. While there are a few benefits, it also creates problems as mentioned above. Since ESM is being used more and more widely, and browsers are making more restricted security checks and requirement, the old Blob based solution is going to be replaced.
- To achieve the requirement, specifically, the CSP environment support, we have to offer a non Blob based solution. Therefore, we have to distribute multiple files and drop the single file solution.
Do not run parser/postprocess on Emscripten generated JavaScript.
- Emscripten is evolving quickly so we should only depends on what's in its documentation instead of a certain implementation details. (for example, currently we patch on its code to deal with a special variable _scriptDir)
- Keep the generated files as-is also helps to:
  - reduce the size of ort.min.js
  - make it easier to replace build artifacts when in development/debug
Drop support for non-SIMD and non-MultiThread. This helps to reduce the number of artifacts in distribution.
- (fixed-sized) SIMD is supported in any mainstream JS environment.
- Multi-thread as WebAssembly feature is supported in any mainstream JS environment. In some environment the feature is guarded with cross origin policy, but it can still work if not trying to create any worker.
Use ESM output for Emscripten generated JavaScript.
- There are 2 ways to dynamically import classic (umd) modules and neither of them are recommended:
  - dynamically creating a <script> tag. This changes the HTML structure and have quite a lot of compatibility issue
  - use fetch() and eval(). However eval is strongly suggested to be avoid because there is a great perf hit.
- importing ESM is super easy - just use the import() call. Considering ESM is widely supported in modern browsers and Node.js this is the better option.
Add Blob based solution as a fallback for cross-origin workers.
- There are still wide use case of importing onnxruntime-web from CDN. In this usage, make it able create worker by using fetch()+Blob to create a same-origin Blob URL.

Distribution File Manifest

The distribution folder contains the following files:

WebAssembly artifacts. These files are the result of compiling the ONNX Runtime C++ code to WebAssembly by Emscripten.

File Name	Build Flags
ort-wasm-simd-threaded.mjs ort-wasm-simd-threaded.wasm	`--enable_wasm_simd` `--enable_wasm_threads`
ort-training-wasm-simd-threaded.mjs ort-training-wasm-simd-threaded.wasm	`--enable_training_apis` `--enable_wasm_simd` `--enable_wasm_threads`
ort-wasm-simd-threaded.jsep.mjs ort-wasm-simd-threaded.jsep.wasm	`--enable_wasm_simd` `--enable_wasm_threads` `--use_jsep` `--use_webnn`

onnxruntime-web JavaScript artifacts. These files are generated by ESBuild as the entry point for onnxruntime-web.

There are multiple build targets for different use cases:

Target Name	Path for "import" or "require"	Description
`ort`	`onnxruntime-web`	The default target.
`ort.all`	`onnxruntime-web/all`	The target including webgl.
`ort.node`	`onnxruntime-web`	The default target for Node.js.
`ort.training`	`onnxruntime-web/training`	The target including training APIs
`ort.wasm`	`onnxruntime-web/wasm`	The target including only WebAssembly (CPU) EP
`ort.webgl`	`onnxruntime-web/webgl`	The target including only WebGL EP

For each target, there are multiple files generated:

File Name	Description
[target].js	The entry point for the target. IIFE and CommonJS format.
[target].mjs	The entry point for the target. ESM format.
[target].min.js [target].min.js.map	The entry point for the target. Minimized with sourcemap. IIFE and CommonJS format.
[target].min.mjs [target].min.mjs.map	The entry point for the target. Minimized with sourcemap. ESM format.
[target].proxy.mjs	(if appliable) The proxy ESM module for the target.
[target].proxy.min.mjs [target].proxy.min.mjs.map	(if appliable) The proxy ESM module for the target. Minimized with sourcemap.

Dynamic Import Explained

Local Served | No Proxy:

[Bundle or ort.min.js]
  |
  + import()--> [ort-wasm-simd-threaded.mjs]
                  |
                  + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm]
                  |
                  + new Worker()--> [ort-wasm-simd-threaded.mjs (worker)]
                                      |
                                      + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm]

Local Served | Proxy:

[Bundle or ort.min.js]
  |
  + import()--> [ort.proxy.min.mjs]
                  |
                  + new Worker()--> [ort.proxy.min.mjs (worker)]
                                      |
                                      + import()--> [ort-wasm-simd-threaded.mjs]
                                                      |
                                                      + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm]
                                                      |
                                                      + new Worker()--> [ort-wasm-simd-threaded.mjs (worker)]
                                                                          |
                                                                          + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm]

Cross Origin | No Proxy:

[Bundle or ort.min.js]
  |
  + fetch('ort-wasm-simd-threaded.mjs')
      |
      + URL.createObjectURL(res.blob())
      |
      + import()--> [blob:... (ort-wasm-simd-threaded)]
                      |
                      + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm]
                      |
                      + new Worker()--> [blob:... (ort-wasm-simd-threaded) (worker)]
                                          |
                                          + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm]

Cross Origin | Proxy

[Bundle or ort.min.js]
  |
  + fetch('ort.proxy.min.mjs')
      |
      + URL.createObjectURL(res.blob())
      |
      + import()--> [blob:... (ort.proxy)]
                      |
                      + new Worker()--> [blob:... (ort.proxy) (worker)]
                                          |
                                          + fetch('ort-wasm-simd-threaded.mjs')
                                              |
                                              + URL.createObjectURL(res.blob())
                                              |
                                              + import()--> [blob:... (ort-wasm-simd-threaded)]
                                                              |
                                                              + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm]
                                                              |
                                                              + new Worker()--> [blob:... (ort-wasm-simd-threaded) (worker)]
                                                                                  |
                                                                                  + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm]

js/web/script/build.ts

js/web/lib/wasm/wasm-utils-import.ts

js/web/lib/wasm/proxy-worker/main.ts

guschmue · 2024-04-02T18:15:52Z

@xenova, any feedback or insight is greatly appreciated

fs-eire · 2024-04-03T05:59:51Z

It seems that import.meta.url will be rewritten into a local path (file://) in some bundlers (webpack, parcel). In this change, onnxruntime-web need import.meta.url to be evaluated at runtime so this behavior breaks the execution.

In webpack we can use importMeta: false in config to skip evaluating import.meta.

~~In Parcel, it seems no way to change this behavior (see this issue). So we may have to consider not to support Parcel.~~

EDIT (2024-04-28): Used a post-process script to workaround this.

js/web/lib/wasm/wasm-utils-import.ts

pranavsharma

Approving for admin

davazp · 2024-08-21T14:02:12Z

In webpack we can use importMeta: false in config to skip evaluating import.meta.

Is there a workaround for applications that do not produce esm modules for the browser? even if we disable the evaluation of import.meta in webpack, the browser will complain about using importa.meta in a non-module.

fs-eire · 2024-08-22T04:53:43Z

If your project is not using ESM, webpack should be picking up the ort{...}.min.js instead of ort{...}.min.mjs. The file ort{...}.min.js should not contain any import.meta expressions.

Please let us know if you see anything unexpected and share the reproduce step so we can try to debug.

davazp · 2024-08-22T10:20:36Z

I think the problem is the project uses ESM as input format but it does not produce ESM as an output for the browser. The import condition is still used from package.json.

I forced it to use the .js files instead of the .mjs and I could make it work, though webpack is complaining that cannot statically determine what files will be imported. It is a warning though, and copying the files to the output manually will work. It does require quite a bit of setup.

I think it would be better if I can continue using the import, but have more control over the path that will be used in the new Worker(), or the worker initialization itself. I will play with the wasmPaths and wasmBinary options and report back later. Thanks for the help!

davazp · 2024-08-22T11:19:08Z

Note the new Worker() error is not on the wasm file but the .mjs actually. I do not think I can fix it with wasmPaths / wasmBinary:

no available backend found. ERR: [webgpu] SecurityError: Failed to construct 'Worker': Script at 'file:///Users/davazp/.../node_modules/onnxruntime-web/dist/ort.webgpu.bundle.min.mjs' cannot be accessed from origin 'https://...'.

fs-eire · 2024-08-26T03:20:59Z

Note the new Worker() error is not on the wasm file but the .mjs actually. I do not think I can fix it with wasmPaths / wasmBinary:
no available backend found. ERR: [webgpu] SecurityError: Failed to construct 'Worker': Script at 'file:///Users/davazp/.../node_modules/onnxruntime-web/dist/ort.webgpu.bundle.min.mjs' cannot be accessed from origin 'https://...'.

Setting importMeta: false is used to exactly fix this problem.

davazp · 2024-08-26T07:41:12Z

Setting importMeta: false is used to exactly fix this problem

Yes, my observation was about not being able able to specify wasmBinaries for it because it is not the .wasm file the problem.

Recap

A bit of a summary of what the issue is for me.

I can import either .js or the .mjs files. Webpack will try to load .mjs from package.json exports because webpack reads ESM modules.
I need to setup importMeta: false to prevent webpack from replacing import.meta with the local file path.
Webpack will emit a bundled file. When loading it, the browser will complain that import.meta cannot be used outside of ESM modules.

I will delay the update to the latest version until I have more time to look into the webpack configuration. I think some options are:

Configure webpack to emit ESM modules. Is it possible?
Force webpack to load the .js (not the .mjs) for this dependency only and ignore the warning of "unable to detect dependencies statically"

Again, thanks for the help!

fs-eire · 2024-08-27T04:47:47Z

I can import either .js or the .mjs files. Webpack will try to load .mjs from package.json exports because webpack reads ESM modules.

Webpack looks at a few different places to determine whether to load a library as UMD or ESM:

whether your package.json contains "type": "module"
whether your source code has .mjs extension instead of .js (for typescript, .mts/.ts)
whether you have explicitly set loading library as ESM for onnxruntime-web in your webpack.config.js

You may figure out that webpack may look at more places because there are various of use scenarios. Anyway, in your case, it decides to load onnxruntime-web as ESM.

It's good so far.

I need to setup importMeta: false to prevent webpack from replacing import.meta with the local file path.

Yes this is expected. It's good so far.

Webpack will emit a bundled file. When loading it, the browser will complain that import.meta cannot be used outside of ESM modules.

This is the problem. So far, the information does not tell whether the webpack is trying to create a UMD bundle or ESM bundle. I guess webpack tries to emit a ESM bundle. If so, when browser complain about that error message, you should make sure that your web app uses the bundle in ESM.

Specifically, your HTML should include the bundle like this:

<script src="your.bundle.min.js" type="module"></script>

Please note the type="module" part.

If Webpack tries to load dependency as ESM but output UMD.... This does not work for onnxruntime-web. It's not common that webpack uses different format for loading dependencies and output bundle.

Configure webpack to emit ESM modules. Is it possible?

Check the file change of webpack.config.js here -> https://github.com/ggaabe/extension/pull/1/files

Force webpack to load the .js (not the .mjs) for this dependency only and ignore the warning of "unable to detect dependencies statically"

I am pretty sure Webpack can do both of them. The problem is the webpack documentation is amazing and it might take a while before you find the correct way to write the config. Good luck!

fs-eire added 9 commits March 26, 2024 15:32

w

ddde964

w2

972a25b

w3

add9d91

w4

551a52e

w5

dcb33da

w6

40dc969

w7

2a554a1

w8

1fba1db

w9

e510b79

github-advanced-security bot found potential problems Apr 1, 2024

View reviewed changes

js/web/script/build.ts Dismissed Show dismissed Hide dismissed

js/web/script/build.ts Dismissed Show dismissed Hide dismissed

js/web/lib/wasm/wasm-utils-import.ts Fixed Show fixed Hide fixed

js/web/lib/wasm/proxy-worker/main.ts Dismissed Show dismissed Hide dismissed

fs-eire added 2 commits April 1, 2024 13:50

fix http-server not close

7ab14ad

resolve one comment

3b74efb

fs-eire changed the title ~~[WIP][js/web] optimize module export and distribution~~ [WIP][js/web] optimize module export and deployment Apr 1, 2024

fs-eire added 2 commits April 2, 2024 01:19

fix e2e test

6fe7416

Merge remote-tracking branch 'origin/main' into fs-eire/wasm-es6

ecb18da

fs-eire marked this pull request as ready for review April 2, 2024 08:20

fs-eire requested a review from a team as a code owner April 2, 2024 08:20

fs-eire changed the title ~~[WIP][js/web] optimize module export and deployment~~ [js/web] optimize module export and deployment Apr 2, 2024

fs-eire added 2 commits April 2, 2024 15:27

parcel

061c237

fix source url

3c00c63

use post-process script

20c54f8

fs-eire force-pushed the fs-eire/wasm-es6 branch from 6a6383f to 20c54f8 Compare April 3, 2024 17:27

fs-eire added 2 commits April 7, 2024 21:04

Merge remote-tracking branch 'origin/main' into fs-eire/wasm-es6

42a6674

fix bundle test

283f5fe

fs-eire mentioned this pull request Apr 9, 2024

Issue 19708: Allow override scripts for threaded worker path #19771

Open

fs-eire added 3 commits April 10, 2024 12:09

Merge remote-tracking branch 'origin/main' into fs-eire/wasm-es6

51ff5a9

Merge remote-tracking branch 'origin/main' into fs-eire/wasm-es6

c32f7d0

fix script

fe780db

Merge remote-tracking branch 'origin/main' into fs-eire/wasm-es6-test

48b570b

fs-eire requested a review from a team as a code owner April 30, 2024 05:01

fs-eire added 7 commits April 29, 2024 22:33

a few build script update

c7edbcc

upgrade emsdk to 3.1.59

1644c06

allow to bundle modules in ESM outputs

7fb3c59

Merge remote-tracking branch 'origin/main' into fs-eire/wasm-es6

b1b36e1

Merge remote-tracking branch 'origin/main' into fs-eire/wasm-es6

e953b65

use original artifact

9531fe8

Merge remote-tracking branch 'origin/main' into fs-eire/wasm-es6

35b277c

github-advanced-security bot found potential problems May 12, 2024

View reviewed changes

js/web/lib/wasm/wasm-utils-import.ts Dismissed Show dismissed Hide dismissed

fs-eire added 5 commits May 13, 2024 22:49

always bundle proxy and never bundle emscripten glue

611ca34

Merge remote-tracking branch 'origin/main' into fs-eire/wasm-es6

4c60105

small optimizations

a16cd2b

remove unused comments

d1ea5e0

Merge remote-tracking branch 'origin/main' into fs-eire/wasm-es6

00b2833

guschmue approved these changes May 17, 2024

View reviewed changes

pranavsharma approved these changes May 17, 2024

View reviewed changes

snnn approved these changes May 20, 2024

View reviewed changes

snnn merged commit 036fcd9 into main May 20, 2024
94 of 96 checks passed

snnn deleted the fs-eire/wasm-es6 branch May 20, 2024 16:51

guschmue mentioned this pull request May 23, 2024

[Web] chrome V3 extension TypeError: URL.createObjectURL is not a function #14445

Closed

kungfooman mentioned this pull request Aug 6, 2024

update the onnxruntime-web version to support the multithreads huggingface/transformers.js#882

Open

gyagp mentioned this pull request Sep 2, 2024

The "path" argument must be of type string or an instance of URL. Received undefined huggingface/transformers.js#916

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[js/web] optimize module export and deployment #20165

[js/web] optimize module export and deployment #20165

fs-eire commented Apr 1, 2024 •

edited

Loading

guschmue commented Apr 2, 2024

fs-eire commented Apr 3, 2024 •

edited

Loading

pranavsharma left a comment

davazp commented Aug 21, 2024

fs-eire commented Aug 22, 2024 •

edited

Loading

davazp commented Aug 22, 2024

davazp commented Aug 22, 2024

fs-eire commented Aug 26, 2024 •

edited

Loading

davazp commented Aug 26, 2024

fs-eire commented Aug 27, 2024 •

edited

Loading

[js/web] optimize module export and deployment #20165

[js/web] optimize module export and deployment #20165

Conversation

fs-eire commented Apr 1, 2024 • edited Loading

Description

Preview

Breaking changes

Importing:

Current import table:

Proposed update:

Flags:

Bundler compatibility:

Deployment:

Problems

Goals

Important Design Decisions

Distribution File Manifest

Dynamic Import Explained

guschmue commented Apr 2, 2024

fs-eire commented Apr 3, 2024 • edited Loading

pranavsharma left a comment

Choose a reason for hiding this comment

davazp commented Aug 21, 2024

fs-eire commented Aug 22, 2024 • edited Loading

davazp commented Aug 22, 2024

davazp commented Aug 22, 2024

fs-eire commented Aug 26, 2024 • edited Loading

davazp commented Aug 26, 2024

Recap

fs-eire commented Aug 27, 2024 • edited Loading

fs-eire commented Apr 1, 2024 •

edited

Loading

fs-eire commented Apr 3, 2024 •

edited

Loading

fs-eire commented Aug 22, 2024 •

edited

Loading

fs-eire commented Aug 26, 2024 •

edited

Loading

fs-eire commented Aug 27, 2024 •

edited

Loading