chore: run on rspack #1

hardfist · 2024-08-06T11:47:46Z

No description provided.

hardfist · 2024-08-06T11:49:58Z

patches/@docusaurus__core@3.4.0.patch

+     const chunkNamesNeeded = matches.flatMap((match) => getChunkNamesToLoad(match.route.path));
+     // Prefetch all webpack chunk assets file needed.
+     return Promise.all(chunkNamesNeeded.map((chunkName) => {
+-        // "__webpack_require__.gca" is injected by ChunkAssetPlugin. Pass it


ChunkAssetPlugin not working now

hardfist · 2024-08-06T12:02:57Z

patches/@docusaurus__core@3.4.0.patch

+ const fs_extra_1 = tslib_1.__importDefault(require("fs-extra"));
+ const path_1 = tslib_1.__importDefault(require("path"));
+-const mini_css_extract_plugin_1 = tslib_1.__importDefault(require("mini-css-extract-plugin"));
+const mini_css_extract_plugin = require('webpack').CssExtractRspackPlugin;


use rspack's builtin cssExtractRspackPlugin

hardfist · 2024-08-06T12:13:53Z

patches/@docusaurus__theme-classic@3.4.0.patch

+                     // This allows better optimization by only bundling those components
+                     // that the user actually needs, because the modules are dynamically
+                     // required and can't be known during compile time.
+-                    new ContextReplacementPlugin(/prismjs[\\/]components$/, new RegExp(`^./(${prismLanguages})$`)),


ContextReplacementPlugin is not supported yet, tracked here web-infra-dev/rspack#7474

slorber · 2024-08-14T11:51:36Z

Thanks @hardfist , I created an initial Docusaurus POC PR here: facebook/docusaurus#10402

There are more things to fix, but our own website (more complex than the init template) is indeed able to run in dev mode.

Our prod build fails with:

[PERF] Build > en > Creating webpack configs - 240.06 ms
[ERROR] Error:   × TypeError: Cannot read properties of undefined (reading 'set')
  │     at Object.fn (/Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/webpack/lib/DefinePlugin.js:335:37)
  │     at SyncHook.callAsyncStageRange (/Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/@rspack/lite-tapable/dist/index.js:240:21)
  │     at SyncHook.callStageRange (/Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/@rspack/lite-tapable/dist/index.js:259:14)
  │     at QueriedHook.call (/Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/@rspack/lite-tapable/dist/index.js:218:26)
  │     at /Users/sebastienlorber/Desktop/projects/docusaurus/packages/docusaurus/node_modules/webpack/dist/Compiler.js:574:244
  │     at last.function (/Users/sebastienlorber/Desktop/projects/docusaurus/packages/docusaurus/node_modules/webpack/dist/Compiler.js:815:28)
  │


[ERROR] Error: Unable to build website for locale en.
    at tryToBuildLocale (/Users/sebastienlorber/Desktop/projects/docusaurus/packages/docusaurus/lib/commands/build.js:54:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/sebastienlorber/Desktop/projects/docusaurus/packages/docusaurus/lib/commands/build.js:65:9
    at async mapAsyncSequential (/Users/sebastienlorber/Desktop/projects/docusaurus/packages/docusaurus-utils/lib/jsUtils.js:21:24)
    at async Object.async (/Users/sebastienlorber/Desktop/projects/docusaurus/packages/docusaurus/lib/utils.js:66:24)
    at async Command.build (/Users/sebastienlorber/Desktop/projects/docusaurus/packages/docusaurus/lib/commands/build.js:63:5) {
  [cause]: [Error:   × TypeError: Cannot read properties of undefined (reading 'set')
    │     at Object.fn (/Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/webpack/lib/DefinePlugin.js:335:37)
    │     at SyncHook.callAsyncStageRange (/Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/@rspack/lite-tapable/dist/index.js:240:21)
    │     at SyncHook.callStageRange (/Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/@rspack/lite-tapable/dist/index.js:259:14)
    │     at QueriedHook.call (/Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/@rspack/lite-tapable/dist/index.js:218:26)
    │     at /Users/sebastienlorber/Desktop/projects/docusaurus/packages/docusaurus/node_modules/webpack/dist/Compiler.js:574:244
    │     at last.function (/Users/sebastienlorber/Desktop/projects/docusaurus/packages/docusaurus/node_modules/webpack/dist/Compiler.js:815:28)
    │
  ] {
    code: 'GenericFailure'
  }
}

hardfist · 2024-08-14T13:46:15Z

@slorber it seems you use webpack/lib/DefinePlugin other than {webpack|rspack}.DefinePlugin, try use rspack.DefinePlugin instead

we consider compatible with rspack/lib/DefinePlugin but after we discussed with webpack team, the webpack/lib/xxxPlugin is actually a bad pattern(not friendly to monorepo) and maybe deprecated in the future, so we give up compatible with this usage.

slorber · 2024-08-14T15:52:52Z

@hardfist even entirely removing the DefinePlugin usage still lead to that error message 🤪 not sure what is going on here but will try to find.

facebook/docusaurus@ca7fefc

slorber · 2024-08-14T16:20:05Z

This is what fails:

Of course we also have webpack.EnvironmentPlugin calls 😄

We now have Rspack building the prod website! https://deploy-preview-10402--docusaurus-2.netlify.app/

slorber · 2024-08-14T17:44:45Z

@hardfist some little questions I'm currently asking myself:

Is support for persistent caching progressing?

Currently, it looks like Webpack on warm cache builds faster than Rspack on cold cache, so not having this will probably be annoying for our users who leverage persistent caching.

How can I know what's slow in the bundling phase? Do you know tools in Webpack/Rspack I can use to profile bundling steps?

I suspect the MDX loader is what takes time (considering we have many MDX docs to compile/bundle per site) but I have no proof of it. I'd like to know how much effort I should put into adopting mdx-rs.

Also wondering how Rspack can parallelize bundling and loader work across multiple cores. Are there settings to fine-tune or does it happen automatically?

Wondering if we should try to use layers to build even faster?

https://x.com/rspack_dev/status/1818490358319530221

For now, we build with compile([clientConfig, serverConfig]) (and we may add a RSC target sooner or later).

Can using layers make it faster?

hardfist · 2024-08-14T19:04:18Z

@slorber

How can I know what's slow in the bundling phase? Do you know tools in Webpack/Rspack I can use to profile bundling steps?

you can generate profile by following this guide https://rspack.dev/guide/optimization/profile, I think it's mostly caused by babel-loader(which should be replaces with builtin:swc-loader)

Is support for persistent caching progressing?

yes we're working on persistent cache and may land in next quarter

Wondering if we should try to use layers to build even faster?

layer is unlikely speedup the build a lot and I'm gonna help improve performance tomorrow

slorber · 2024-08-15T11:44:51Z

Thanks

you can generate profile by following this guide rspack.dev/guide/optimization/profile, I think it's mostly caused by babel-loader(which should be replaces with builtin:swc-loader)

Looks like RSPACK_PROFILE only works with the CLI right? Is it useful to compliment Rsdoctor or Rsdoctor is enough?

I got it working but is there a way to save the Rsdoctor result somewhere?

Is support for persistent caching progressing?

yes we're working on persistent cache and may land in next quarter

Great to know 🔥!

Wondering if we should try to use layers to build even faster?

layer is unlikely speedup the build a lot and I'm gonna help improve performance tomorrow

Thanks 🙏

Not sure it's related but there's also one thing that's really annoying me in our current setup: we have a lot of MDX documents and they are compiled by MDX twice, by the server and the client bundler config, using our mdx loader.

These documents are unlikely to be compiled differently between server/client. Is there a way to share the compilation result between both configs so that it only runs once for both envs?

hardfist · 2024-08-16T01:47:08Z

Not sure it's related but there's also one thing that's really annoying me in our current setup: we have a lot of MDX documents and they are compiled by MDX twice, by the server and the client bundler config, using our mdx loader.

Layer actually be helpful here, but it may involve need big refactor for docusaurus, and I think a cross compiler cache for mdx-loader could help here?

Looks like RSPACK_PROFILE only works with the CLI right? Is it useful to compliment Rsdoctor or Rsdoctor is enough?

yes, but we provide node api for this, rsdoctor don't support provide rust performance yet.

hardfist · 2024-08-16T08:05:17Z

@slorber I did some profile and find some bottleneck
CPU.20240816.154753.69358.0.001.cpuprofile

html-minifier-tersser: 300ms, could be replace with rspackHtmlPlugin
mdx-loader: cost 3s here
pwa-plugin: 1s

slorber · 2024-08-16T15:36:48Z

Layer actually be helpful here, but it may involve need big refactor for docusaurus, and I think a cross compiler cache for mdx-loader could help here?

Yes sharing an MDX compilation cache between the 2 compilers/loaders would give great results.

Is there a first-class solution to implement this in Webpack/Rspack, or is it something to implement in userland? I could give it a try but I'm afraid of introducing cache invalidation problems 🤪

@slorber I did some profile and find some bottleneck CPU.20240816.154753.69358.0.001.cpuprofile

max-loader: cost 3s here

I assume you mean "mdx-loader" and not "max-loader"? I looked into your mdx-rs fork but I think we can't go all-in on Rust and need to keep support for js remark plugins (at least until I'm able to port all our plugins to Rust, but I don't know Rust yet 😅). I'd like to see if I can't use Rust only for parsing to AST, because it looks like the most expensive part (but mdx-rs doesn't support parsing directives yet so I might start my Rust journey here 🤪 ).

pwa-plugin: 1s

The PWA plugin is not critical to optimize, it's not widely used.

html-minifier-tersser: 300ms, could be replace with rspackHtmlPlugin

html-minifier-terser I see much more than 300ms on your profile

rspackHtmlPlugin does not look like a replacement for html-minifier-terser, it does not minify by default. Afaik @chenjiahan minification pluggable and the example shows usage of html-minifier-terser, which is probably the slow part taking a few seconds.

We don't currently do the SSG as part of the Webpack bundling. Do you think it would bring any benefit? I'd prefer to keep this part decoupled from bundler code unless there's a good reason to couple it.

It doesn't look like Rspress generate pages as part of the Rspack build either, and it doesn't look like it minifies the SSG HTML files:

https://github.com/web-infra-dev/rspress/blob/main/packages/core/src/node/build.ts#L142C18-L142C25

hardfist · 2024-08-16T15:43:38Z

@slorber https://rspack.dev/plugins/rspack/html-rspack-plugin did support minify by default and use swc-html-minify to the minify work so it's much faster
yeah ssg is doing in separa parts and can be disabled, i don't see the benefits of doing ssg during bundling, but it seems server bundle should be faster than client bundle cause it doesn't need to be minified and can external tons of node_modules, but it seems docusaurus spend near same for client & server bundle, if server bundle can finish earlier then it can do ssg parts during client bundle, which may reduce the total time

slorber · 2024-08-16T16:13:14Z

@slorber rspack.dev/plugins/rspack/html-rspack-plugin did support minify by default and use swc-html-minify to the minify work so it's much faster

Do you have a reference to the code?

Is this that rust crate? And are there JS bindings for it?
https://github.com/swc-project/swc/tree/main/crates/swc_html_minifier

What I need is function minify(html: string): Promise<string>

yeah ssg is doing in separa parts and can be disabled, i don't see the benefits of doing ssg during bundling, but it seems server bundle should be faster than client bundle cause it doesn't need to be minified

As far as I remember we don't minify the server bundle.

and can external tons of node_modules,

Regarding externals, I'll think about it but historically we isolate SSG in a Node vm.Script, maybe that's the reason. I will need to test a few things here, it has always been like that and maybe we can optimize.

but it seems docusaurus spend near same for client & server bundle, if server bundle can finish earlier then it can do ssg parts during client bundle, which may reduce the total time

That's something I thought about, but currently, the SSG process needs a manifest that is generated as part of the client bundling. As far as I remember this is used to avoid waterfalls and be able to load all the required chunks up-front on initial load and link hover.

All the client chunks to load are inlined in the SSG html files. Quite similar to this technique: https://mmazzarolo.com/blog/2024-08-13-async-chunk-preloading-on-load/

Maybe a solution could be to bundle the app on client/server in a way that would generate the exact same manifest, so that we could start the SSG using the server compiler manifest. Worth trying.

hardfist · 2024-08-16T16:37:07Z

Do you have a reference to the code?

we do it in rust side https://github.com/web-infra-dev/rspack/blob/main/crates/rspack_plugin_html/src/plugin.rs#L56

Is this that rust crate? And are there JS bindings for it?
swc-project/swc@main/crates/swc_html_minifier
What I need is function minify(html: string): Promise

it seems no one made js binding for it even it's simple

hardfist added 3 commits August 6, 2024 19:23

chore: redirect webpack to rspack

8c906b8

chore: patch contextReplacementPlugin

5ff901d

chore: add loose mode

5965800

hardfist commented Aug 6, 2024

View reviewed changes

hardfist force-pushed the rspack branch 2 times, most recently from 137c3f4 to 66b2863 Compare August 6, 2024 12:02

hardfist commented Aug 6, 2024

View reviewed changes

hardfist force-pushed the rspack branch from 66b2863 to 144efdd Compare August 6, 2024 12:10

chore: patch @docusaurus/core

cdd6297

hardfist force-pushed the rspack branch from 144efdd to cdd6297 Compare August 6, 2024 12:11

hardfist commented Aug 6, 2024

View reviewed changes

slorber mentioned this pull request Aug 14, 2024

feat(core): Replace Webpack with Rspack - siteConfig.future.experimental_faster.rspackBundler facebook/docusaurus#10402

Merged

ruibaby mentioned this pull request Oct 17, 2024

chore: bump docusaurus to canary verison halo-dev/docs#428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: run on rspack #1

chore: run on rspack #1

hardfist commented Aug 6, 2024

hardfist Aug 6, 2024

hardfist Aug 6, 2024

hardfist Aug 6, 2024

slorber commented Aug 14, 2024

hardfist commented Aug 14, 2024 •

edited

Loading

slorber commented Aug 14, 2024

slorber commented Aug 14, 2024 •

edited

Loading

slorber commented Aug 14, 2024

hardfist commented Aug 14, 2024 •

edited

Loading

slorber commented Aug 15, 2024

hardfist commented Aug 16, 2024 •

edited

Loading

hardfist commented Aug 16, 2024 •

edited

Loading

slorber commented Aug 16, 2024

hardfist commented Aug 16, 2024 •

edited

Loading

slorber commented Aug 16, 2024

hardfist commented Aug 16, 2024

chore: run on rspack #1

Are you sure you want to change the base?

chore: run on rspack #1

Conversation

hardfist commented Aug 6, 2024

hardfist Aug 6, 2024

Choose a reason for hiding this comment

hardfist Aug 6, 2024

Choose a reason for hiding this comment

hardfist Aug 6, 2024

Choose a reason for hiding this comment

slorber commented Aug 14, 2024

hardfist commented Aug 14, 2024 • edited Loading

slorber commented Aug 14, 2024

slorber commented Aug 14, 2024 • edited Loading

slorber commented Aug 14, 2024

hardfist commented Aug 14, 2024 • edited Loading

slorber commented Aug 15, 2024

hardfist commented Aug 16, 2024 • edited Loading

hardfist commented Aug 16, 2024 • edited Loading

slorber commented Aug 16, 2024

hardfist commented Aug 16, 2024 • edited Loading

slorber commented Aug 16, 2024

hardfist commented Aug 16, 2024

hardfist commented Aug 14, 2024 •

edited

Loading

slorber commented Aug 14, 2024 •

edited

Loading

hardfist commented Aug 14, 2024 •

edited

Loading

hardfist commented Aug 16, 2024 •

edited

Loading

hardfist commented Aug 16, 2024 •

edited

Loading

hardfist commented Aug 16, 2024 •

edited

Loading