Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve image optimizer to only create 1 worker thread #23188

Merged
merged 3 commits into from
Mar 18, 2021
Merged

Improve image optimizer to only create 1 worker thread #23188

merged 3 commits into from
Mar 18, 2021

Conversation

shuding
Copy link
Member

@shuding shuding commented Mar 18, 2021 •
edited
Loading

As stated in #23157, this PR merges all the operations into 1 worker thread (processBuffer in impl.ts) and only pass a list of operation names and args into the worker. This should improve the speed and memory usage of next/image.

Fixes #23157. X-ref: #22925.

@ijjk
Copy link
Member

ijjk commented Mar 18, 2021

Stats from current PR

Default Server Mode (Increase detected ⚠️)
General Overall decrease ✓
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
buildDuration 13.5s 13.2s -283ms
nodeModulesSize 43.6 MB 43.6 MB -979 B
Page Load Tests Overall increase ✓
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
/ failed reqs 0 0 ✓
/ total time (seconds) 2.386 2.359 -0.03
/ avg req/sec 1047.83 1059.75 +11.92
/error-in-render failed reqs 0 0 ✓
/error-in-render total time (seconds) 1.605 1.601 0
/error-in-render avg req/sec 1557.48 1561.51 +4.03
Client Bundles (main, webpack, commons)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
677f882d2ed8..a2e7.js gzip 13.4 kB 13.4 kB ✓
framework.HASH.js gzip 39 kB 39 kB ✓
main-HASH.js gzip 7.14 kB 7.14 kB ✓
webpack-HASH.js gzip 751 B 751 B ✓
Overall change 60.3 kB 60.3 kB ✓
Legacy Client Bundles (polyfills)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
polyfills-HASH.js gzip 31.3 kB 31.3 kB ✓
Overall change 31.3 kB 31.3 kB ✓
Client Pages
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_app-fde3324..9dd1.js gzip 1.28 kB 1.28 kB ✓
_error-af59f..582f.js gzip 3.46 kB 3.46 kB ✓
amp-9716187d..0aa8.js gzip 536 B 536 B ✓
hooks-107e90..74c7.js gzip 888 B 888 B ✓
index-ac435c..ecf2.js gzip 227 B 227 B ✓
link-e6cee42..49eb.js gzip 1.64 kB 1.64 kB ✓
routerDirect..dc9d.js gzip 303 B 303 B ✓
withRouter-6..0e02.js gzip 302 B 302 B ✓
Overall change 8.64 kB 8.64 kB ✓
Client Build Manifests
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_buildManifest.js gzip 346 B 346 B ✓
Overall change 346 B 346 B ✓
Rendered Page Sizes
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
index.html gzip 612 B 612 B ✓
link.html gzip 621 B 621 B ✓
withRouter.html gzip 607 B 607 B ✓
Overall change 1.84 kB 1.84 kB ✓
Serverless Mode (Decrease detected ✓)
General Overall decrease ✓
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
buildDuration 15s 15.5s ⚠️ +416ms
nodeModulesSize 43.6 MB 43.6 MB -979 B
Client Bundles (main, webpack, commons)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
677f882d2ed8..a2e7.js gzip 13.4 kB 13.4 kB ✓
framework.HASH.js gzip 39 kB 39 kB ✓
main-HASH.js gzip 7.14 kB 7.14 kB ✓
webpack-HASH.js gzip 751 B 751 B ✓
Overall change 60.3 kB 60.3 kB ✓
Legacy Client Bundles (polyfills)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
polyfills-HASH.js gzip 31.3 kB 31.3 kB ✓
Overall change 31.3 kB 31.3 kB ✓
Client Pages
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_app-fde3324..9dd1.js gzip 1.28 kB 1.28 kB ✓
_error-af59f..582f.js gzip 3.46 kB 3.46 kB ✓
amp-9716187d..0aa8.js gzip 536 B 536 B ✓
hooks-107e90..74c7.js gzip 888 B 888 B ✓
index-ac435c..ecf2.js gzip 227 B 227 B ✓
link-e6cee42..49eb.js gzip 1.64 kB 1.64 kB ✓
routerDirect..dc9d.js gzip 303 B 303 B ✓
withRouter-6..0e02.js gzip 302 B 302 B ✓
Overall change 8.64 kB 8.64 kB ✓
Client Build Manifests
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_buildManifest.js gzip 346 B 346 B ✓
Overall change 346 B 346 B ✓
Serverless bundles
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_error.js 1.02 MB 1.02 MB ✓
404.html 2.67 kB 2.67 kB ✓
500.html 2.65 kB 2.65 kB ✓
amp.amp.html 10.6 kB 10.6 kB ✓
amp.html 1.86 kB 1.86 kB ✓
hooks.html 1.92 kB 1.92 kB ✓
index.js 1.02 MB 1.02 MB ✓
link.js 1.08 MB 1.08 MB ✓
routerDirect.js 1.07 MB 1.07 MB ✓
withRouter.js 1.07 MB 1.07 MB ✓
Overall change 5.27 MB 5.27 MB ✓
Webpack 5 Mode (Decrease detected ✓)
General Overall decrease ✓
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
buildDuration 16s 15.5s -510ms
nodeModulesSize 43.6 MB 43.6 MB -979 B
Page Load Tests Overall decrease ⚠️
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
/ failed reqs 0 0 ✓
/ total time (seconds) 2.347 2.398 ⚠️ +0.05
/ avg req/sec 1065.2 1042.53 ⚠️ -22.67
/error-in-render failed reqs 0 0 ✓
/error-in-render total time (seconds) 1.549 1.68 ⚠️ +0.13
/error-in-render avg req/sec 1613.72 1487.83 ⚠️ -125.89
Client Bundles (main, webpack, commons)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
597-12fee761..569d.js gzip 13.3 kB 13.3 kB ✓
framework.HASH.js gzip 39.3 kB 39.3 kB ✓
main-HASH.js gzip 7.1 kB 7.1 kB ✓
webpack-HASH.js gzip 993 B 993 B ✓
Overall change 60.7 kB 60.7 kB ✓
Legacy Client Bundles (polyfills)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
polyfills-HASH.js gzip 31.1 kB 31.1 kB ✓
Overall change 31.1 kB 31.1 kB ✓
Client Pages
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_app-d0eb668..79ef.js gzip 1.28 kB 1.28 kB ✓
_error-d12ff..6063.js gzip 3.4 kB 3.4 kB ✓
amp-127483e4..9cd8.js gzip 560 B 560 B ✓
hooks-75994b..62d3.js gzip 924 B 924 B ✓
index-ac1ee6..819e.js gzip 243 B 243 B ✓
link-1cd4709..71e5.js gzip 1.65 kB 1.65 kB ✓
routerDirect..4d90.js gzip 333 B 333 B ✓
withRouter-0..60b0.js gzip 329 B 329 B ✓
Overall change 8.72 kB 8.72 kB ✓
Client Build Manifests
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_buildManifest.js gzip 324 B 324 B ✓
Overall change 324 B 324 B ✓
Rendered Page Sizes
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
index.html gzip 588 B 588 B ✓
link.html gzip 595 B 595 B ✓
withRouter.html gzip 581 B 581 B ✓
Overall change 1.76 kB 1.76 kB ✓

Diffs

Diff for index.html 
@@ -43,7 +43,7 @@
         "props": { "pageProps": {} },
         "page": "/",
         "query": {},
-        "buildId": "MVQQdrf7UVVIcyOhcWJIh",
+        "buildId": "KrFHKb80vFqFxCYoV7fm8",
         "isFallback": false,
         "gip": true
       }
@@ -77,11 +77,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/MVQQdrf7UVVIcyOhcWJIh/_buildManifest.js"
+      src="/_next/static/KrFHKb80vFqFxCYoV7fm8/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/MVQQdrf7UVVIcyOhcWJIh/_ssgManifest.js"
+      src="/_next/static/KrFHKb80vFqFxCYoV7fm8/_ssgManifest.js"
       async=""
     ></script>
   </body>
Diff for link.html 
@@ -48,7 +48,7 @@
         "props": { "pageProps": {} },
         "page": "/link",
         "query": {},
-        "buildId": "MVQQdrf7UVVIcyOhcWJIh",
+        "buildId": "KrFHKb80vFqFxCYoV7fm8",
         "isFallback": false,
         "gip": true
       }
@@ -82,11 +82,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/MVQQdrf7UVVIcyOhcWJIh/_buildManifest.js"
+      src="/_next/static/KrFHKb80vFqFxCYoV7fm8/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/MVQQdrf7UVVIcyOhcWJIh/_ssgManifest.js"
+      src="/_next/static/KrFHKb80vFqFxCYoV7fm8/_ssgManifest.js"
       async=""
     ></script>
   </body>
Diff for withRouter.html 
@@ -43,7 +43,7 @@
         "props": { "pageProps": {} },
         "page": "/withRouter",
         "query": {},
-        "buildId": "MVQQdrf7UVVIcyOhcWJIh",
+        "buildId": "KrFHKb80vFqFxCYoV7fm8",
         "isFallback": false,
         "gip": true
       }
@@ -77,11 +77,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/MVQQdrf7UVVIcyOhcWJIh/_buildManifest.js"
+      src="/_next/static/KrFHKb80vFqFxCYoV7fm8/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/MVQQdrf7UVVIcyOhcWJIh/_ssgManifest.js"
+      src="/_next/static/KrFHKb80vFqFxCYoV7fm8/_ssgManifest.js"
       async=""
     ></script>
   </body>
Commit: 7c025f1

@shuding shuding marked this pull request as ready for review March 18, 2021 14:03
@ijjk
Copy link
Member

ijjk commented Mar 18, 2021

Stats from current PR

Default Server Mode (Increase detected ⚠️)
General Overall decrease ✓
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
buildDuration 11.6s 11.8s ⚠️ +164ms
nodeModulesSize 43.6 MB 43.6 MB -979 B
Page Load Tests Overall increase ✓
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
/ failed reqs 0 0 ✓
/ total time (seconds) 2.003 2.042 ⚠️ +0.04
/ avg req/sec 1247.92 1224.53 ⚠️ -23.39
/error-in-render failed reqs 0 0 ✓
/error-in-render total time (seconds) 1.264 1.221 -0.04
/error-in-render avg req/sec 1977.94 2047.9 +69.96
Client Bundles (main, webpack, commons)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
677f882d2ed8..a2e7.js gzip 13.4 kB 13.4 kB ✓
framework.HASH.js gzip 39 kB 39 kB ✓
main-HASH.js gzip 7.14 kB 7.14 kB ✓
webpack-HASH.js gzip 751 B 751 B ✓
Overall change 60.3 kB 60.3 kB ✓
Legacy Client Bundles (polyfills)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
polyfills-HASH.js gzip 31.3 kB 31.3 kB ✓
Overall change 31.3 kB 31.3 kB ✓
Client Pages
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_app-fde3324..9dd1.js gzip 1.28 kB 1.28 kB ✓
_error-af59f..582f.js gzip 3.46 kB 3.46 kB ✓
amp-9716187d..0aa8.js gzip 536 B 536 B ✓
hooks-107e90..74c7.js gzip 888 B 888 B ✓
index-ac435c..ecf2.js gzip 227 B 227 B ✓
link-e6cee42..49eb.js gzip 1.64 kB 1.64 kB ✓
routerDirect..dc9d.js gzip 303 B 303 B ✓
withRouter-6..0e02.js gzip 302 B 302 B ✓
Overall change 8.64 kB 8.64 kB ✓
Client Build Manifests
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_buildManifest.js gzip 346 B 346 B ✓
Overall change 346 B 346 B ✓
Rendered Page Sizes
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
index.html gzip 612 B 612 B ✓
link.html gzip 621 B 621 B ✓
withRouter.html gzip 607 B 607 B ✓
Overall change 1.84 kB 1.84 kB ✓
Serverless Mode (Decrease detected ✓)
General Overall decrease ✓
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
buildDuration 13.8s 13.7s -89ms
nodeModulesSize 43.6 MB 43.6 MB -979 B
Client Bundles (main, webpack, commons)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
677f882d2ed8..a2e7.js gzip 13.4 kB 13.4 kB ✓
framework.HASH.js gzip 39 kB 39 kB ✓
main-HASH.js gzip 7.14 kB 7.14 kB ✓
webpack-HASH.js gzip 751 B 751 B ✓
Overall change 60.3 kB 60.3 kB ✓
Legacy Client Bundles (polyfills)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
polyfills-HASH.js gzip 31.3 kB 31.3 kB ✓
Overall change 31.3 kB 31.3 kB ✓
Client Pages
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_app-fde3324..9dd1.js gzip 1.28 kB 1.28 kB ✓
_error-af59f..582f.js gzip 3.46 kB 3.46 kB ✓
amp-9716187d..0aa8.js gzip 536 B 536 B ✓
hooks-107e90..74c7.js gzip 888 B 888 B ✓
index-ac435c..ecf2.js gzip 227 B 227 B ✓
link-e6cee42..49eb.js gzip 1.64 kB 1.64 kB ✓
routerDirect..dc9d.js gzip 303 B 303 B ✓
withRouter-6..0e02.js gzip 302 B 302 B ✓
Overall change 8.64 kB 8.64 kB ✓
Client Build Manifests
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_buildManifest.js gzip 346 B 346 B ✓
Overall change 346 B 346 B ✓
Serverless bundles
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_error.js 1.02 MB 1.02 MB ✓
404.html 2.67 kB 2.67 kB ✓
500.html 2.65 kB 2.65 kB ✓
amp.amp.html 10.6 kB 10.6 kB ✓
amp.html 1.86 kB 1.86 kB ✓
hooks.html 1.92 kB 1.92 kB ✓
index.js 1.02 MB 1.02 MB ✓
link.js 1.08 MB 1.08 MB ✓
routerDirect.js 1.07 MB 1.07 MB ✓
withRouter.js 1.07 MB 1.07 MB ✓
Overall change 5.27 MB 5.27 MB ✓
Webpack 5 Mode (Decrease detected ✓)
General Overall decrease ✓
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
buildDuration 14s 14s ⚠️ +1ms
nodeModulesSize 43.6 MB 43.6 MB -979 B
Page Load Tests Overall increase ✓
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
/ failed reqs 0 0 ✓
/ total time (seconds) 2.027 2.033 ⚠️ +0.01
/ avg req/sec 1233.51 1229.65 ⚠️ -3.86
/error-in-render failed reqs 0 0 ✓
/error-in-render total time (seconds) 1.253 1.248 0
/error-in-render avg req/sec 1995.39 2002.81 +7.42
Client Bundles (main, webpack, commons)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
597-12fee761..569d.js gzip 13.3 kB 13.3 kB ✓
framework.HASH.js gzip 39.3 kB 39.3 kB ✓
main-HASH.js gzip 7.1 kB 7.1 kB ✓
webpack-HASH.js gzip 993 B 993 B ✓
Overall change 60.7 kB 60.7 kB ✓
Legacy Client Bundles (polyfills)
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
polyfills-HASH.js gzip 31.1 kB 31.1 kB ✓
Overall change 31.1 kB 31.1 kB ✓
Client Pages
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_app-d0eb668..79ef.js gzip 1.28 kB 1.28 kB ✓
_error-d12ff..6063.js gzip 3.4 kB 3.4 kB ✓
amp-127483e4..9cd8.js gzip 560 B 560 B ✓
hooks-75994b..62d3.js gzip 924 B 924 B ✓
index-ac1ee6..819e.js gzip 243 B 243 B ✓
link-1cd4709..71e5.js gzip 1.65 kB 1.65 kB ✓
routerDirect..4d90.js gzip 333 B 333 B ✓
withRouter-0..60b0.js gzip 329 B 329 B ✓
Overall change 8.72 kB 8.72 kB ✓
Client Build Manifests
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
_buildManifest.js gzip 324 B 324 B ✓
Overall change 324 B 324 B ✓
Rendered Page Sizes
vercel/next.js canary shuding/next.js feat-optimize-image-optimizer Change
index.html gzip 588 B 588 B ✓
link.html gzip 595 B 595 B ✓
withRouter.html gzip 581 B 581 B ✓
Overall change 1.76 kB 1.76 kB ✓

Diffs

Diff for index.html 
@@ -43,7 +43,7 @@
         "props": { "pageProps": {} },
         "page": "/",
         "query": {},
-        "buildId": "6e0e9sPn97nNs8lI7erlz",
+        "buildId": "JECDLJGDbqx9qqwHwXnbv",
         "isFallback": false,
         "gip": true
       }
@@ -77,11 +77,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/6e0e9sPn97nNs8lI7erlz/_buildManifest.js"
+      src="/_next/static/JECDLJGDbqx9qqwHwXnbv/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/6e0e9sPn97nNs8lI7erlz/_ssgManifest.js"
+      src="/_next/static/JECDLJGDbqx9qqwHwXnbv/_ssgManifest.js"
       async=""
     ></script>
   </body>
Diff for link.html 
@@ -48,7 +48,7 @@
         "props": { "pageProps": {} },
         "page": "/link",
         "query": {},
-        "buildId": "6e0e9sPn97nNs8lI7erlz",
+        "buildId": "JECDLJGDbqx9qqwHwXnbv",
         "isFallback": false,
         "gip": true
       }
@@ -82,11 +82,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/6e0e9sPn97nNs8lI7erlz/_buildManifest.js"
+      src="/_next/static/JECDLJGDbqx9qqwHwXnbv/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/6e0e9sPn97nNs8lI7erlz/_ssgManifest.js"
+      src="/_next/static/JECDLJGDbqx9qqwHwXnbv/_ssgManifest.js"
       async=""
     ></script>
   </body>
Diff for withRouter.html 
@@ -43,7 +43,7 @@
         "props": { "pageProps": {} },
         "page": "/withRouter",
         "query": {},
-        "buildId": "6e0e9sPn97nNs8lI7erlz",
+        "buildId": "JECDLJGDbqx9qqwHwXnbv",
         "isFallback": false,
         "gip": true
       }
@@ -77,11 +77,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/6e0e9sPn97nNs8lI7erlz/_buildManifest.js"
+      src="/_next/static/JECDLJGDbqx9qqwHwXnbv/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/6e0e9sPn97nNs8lI7erlz/_ssgManifest.js"
+      src="/_next/static/JECDLJGDbqx9qqwHwXnbv/_ssgManifest.js"
       async=""
     ></script>
   </body>
Commit: cf8a9d4

@kodiakhq kodiakhq bot merged commit 241a916 into vercel:canary Mar 18, 2021
@shuding shuding deleted the feat-optimize-image-optimizer branch March 18, 2021 16:19
@thedavidprice thedavidprice mentioned this pull request Mar 20, 2021
Closed
@henryStelle henryStelle mentioned this pull request Mar 29, 2021
Closed
@PranavSathy PranavSathy mentioned this pull request Mar 31, 2021
Closed
flybayer pushed a commit to blitz-js/next.js that referenced this pull request Apr 29, 2021
@shuding
Improve image optimizer to only create 1 worker thread (vercel#23188)
8e437ae 
As stated in vercel#23157, this PR merges all the operations into 1 worker thread (`processBuffer` in `impl.ts`) and only pass a list of operation names and args into the worker. This should improve the speed and memory usage of next/image.

Fixes vercel#23157. X-ref: vercel#22925.
@vercel vercel locked as resolved and limited conversation to collaborators Jan 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@timneutkens timneutkens timneutkens approved these changes

@divmain divmain Awaiting requested review from divmain

@ijjk ijjk Awaiting requested review from ijjk

@lfades lfades Awaiting requested review from lfades

@Timer Timer Awaiting requested review from Timer

Assignees
No one assigned
Labels
type: next
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Handle image optimization operations in one worker thread
3 participants
@shuding @ijjk @timneutkens