Automatic tree-shaking of messages

[amannn]

Messages can either be consumed in Server- or Client Components.

The former is recommended in the docs, as it has both performance benefits as well as a simpler model in general since there is only a single environment where code executes. However, for highly interactive apps, using messages on the client side is totally fine too. The situation gets a bit hairy when you need to split messages based on usage in Client Components (e.g. by page, or a mix of RSC / RCC).

Ideally, we'd provide more help with handing off translations to Client Components.

Ideas:

1. User-defined namespaces
2. Manually composed namespaces
3. Compiler-based approach

(1) User-defined namespaces

This is already possible (and shown in the docs), but unfortunately not so maintainable. For single components it can be fine however, so when you have many small interactive leaf components, this is quite ok.

(2) Manually composed namespaces

This worked well for the Pages Router and was documented, but unfortunately doesn't work with RSC.

Apollo has a similar situation with static fragments: apollographql/apollo-client-nextjs#27

A potential solution could be to use a build-time macro that is evaluated separately.

(3) Compiler-based approach

See #1 (comment). Similar to fragment hoisting in Relay, if the collection of namespaces happens at build time, we could provide this list to Server Components. This seems to be the best option.

Design-wise this requires analyzing the namespace that is passed to useTranslations. Furthermore, a nice-to-have would be to analyze which keys are passed to t (if all calls are statically known).

Current progress:

• A proof-of-concept is developed in jantimon/next-poc-i18n-split (based on https://github.com/yashvesikar/next-intl-translation-bundler-example)
• APIs are being researched in https://github.com/amannn/nextjs-compiled-messages (this is a combined experiment that also includes AOT compilation for messages)

See also: Dependency tree crawling

[amannn]

For very performance critical apps, it might make sense to use components that make use of next-intl only on the server side and to provide labels as strings to interactive client components. That way you wouldn't have to tree shake anything.

[nbouvrette]

Thanks @amannn for sharing this with me. For those who might be interested, we built a similar Babel plugin solution that works on Next.js: https://github.com/Avansai/next-multilingual

Of course, as called out, one of the drawbacks is that all locales are loaded in the same bundle which increases the size. I am planning to try to find a solution as we are adding an SWC version of the plugin.

[amannn]

@yashvesikar just shared a POC for a webpack plugin that extracts messages based on page entry points with me: https://github.com/yashvesikar/next-intl-translation-bundler-example

I'm still mind blown 🤯

AFAICT what we'd have to do (assuming we'd implement this for the app directory) is:

1. Detect all entry points. In the POC, every file is treated as an "entry point", meaning that it will emit translations. This would need to be adapted to only handle modules that have the 'use client' banner as these are the boundaries where we're switching to the client side.
2. For every entry point, traverse the module graph and find all references to useTranslations/t and collect the namespaces.
3. Emit the translations per entry point.

The other part is that this is a webpack plugin, no idea if these will be supported by Turbopack or if we'd have to implement it in Rust instead (if Turbopack even supports plugins at this point).

Edit: An improved version of the webpack plugin that supports the App Router was implemented in https://github.com/jantimon/next-poc-i18n-split.

---

Another related project: https://lingui.dev/guides/message-extraction#dependency-tree-crawling-experimental

[yashvesikar]

@amannn Unfortunately, I don't have much knowledge of the app directory and all that RSC entails. I can speak to some of the advancements I implemented to this logic in the past and some of the ones I didn't get to.

• Support for colocated translations by component — Similar to CSS modules. This has a few advantages:
  • IMO it's easier to spot and fix & address translation bugs and errors.
  • Gives the translation plugin full control over the key names and allows the user to use TS type checking! (rough idea) i.e.

// src/components/my-component/i18n/en-US.json
{"title": "Title in English", "other": "Some other Key that won't be bundled"}
// src/components/my-component/i18n/fr.json
{"title": "Title in French"}

// src/components/my-component/i18n/index.ts
// loader can check that the keys match the ones in the main locale json 
let translations = {title: "title"} as const

// src/components/my-component/my-component.tsx
import { translations } from "/.i18n

export const MyComponent() => {
	// by modifying the useTranslations hook we can make t(key) have key be typeof translations
	const t = useTranslations(translations); 
	
	return(
		<div>{t(translations.title)}</div>
	)
}

• Figuring out how to get Webpack to load the translation files as dependencies.
  Right now in the POC, I made the server is loading the files from the disk using fs.readFileSync. This is probably the worst way to read that data for high-traffic scenarios because it is a blocking call. Even making that call async will be much better, even better might be to figure out how to have Webpack resolve the dependency itself such that the translations for a given page/path are available to the getServerSideProps function the same way any other import would be. I spent a little time looking into that but Webpack beat me 😅
  An open question is whether caching would be better done this way or via a separate network request with a long max-age.

• Improving the HMR, right now the HMR in dev kinda works. The POC plugin will build the right translation output files the first time the page is loaded in dev, but if the translation JSON file is changed the server needs to be restarted.

Some known limitations are that the Webpack parser is very limited and fickle. It's not a great way to traverse the AST to get all the keys, but since it's a single toolchain it is nice. I don't have any experience in SWC or TurboPack parsers but I have a hard time imagining they are worse than Webpack.

There are a few more things but I will leave it here for now. Sorry for the wall of text, happy that this may be useful to the project :)

[amannn]

Hey @yashvesikar, thanks a lot for chiming in here and sharing your thoughts in detail!

Support for colocated translations by component

That's an interesting one, I think @nbouvrette is exploring something similar to this.

At least for next-intl, I'm currently not pursuing this direction though and rather try to make it easy to work with message bundles per locale as from my experience this integrates well with using translation management systems. We do have type checking though btw.!

Improving the HMR

Good point! That's something that's working fairly well currently, I'd have to investigate this for sure.

I don't have any experience in SWC or TurboPack parsers but I have a hard time imagining they are worse than Webpack.

😆 Well, I've read that Turbopack is adding support for Webpack loaders, I'm not sure what the story for plugins is though. This topic is generally a bit futuristic at this point and Turbopack is still quite experimental from what I know. At least for the time being, a webpack plugin would definitely be the way to go.

(continuing from our Twitter thread) in nextjs every page is an entrypoint + a main entrypoint for shared assets. so extracting all keys for a page is the same as by entrypoint. the webpack docs are simultaneously really useful and not useful enough 😅

Hmm, I just console.log'd the entry points—and I can confirm the part with the pages being entry points! But e.g. if pages/A.tsx imports components/Profile.tsx, how can I find this reference to process the dependency graph recursively?

Generally I'm currently still quite busy with the RSC integration for next-intl, but your thoughts are really helpful here, I hope to find some time to revisit this topic in the future! Made me laugh the other day when I saw that this was issue #1! 😄

[nbouvrette]

Great to see I'm not the only one wanting to solve this problem. I don't know if you guys saw but I split the next-multilingual logic that dynamically loads message modules into this package: https://github.com/Avansai/messages-modules

It uses Babel to dynamically inject the message in their local scope which is pretty neat but one optimization I would like to fix is that it loads all locales and not just the one being viewed... which of course would end up causing performance issues in the long run. But I think it's still better than a huge message file containing messages that are not even used

The big problem ahead is that from what I understood the app directory cannot use Babel... so basically I will need to rewrite this package in Rust so that it can work with SWC.

If you guys are interested to team up, I would be more than happy to get some help. Otherwise, I will most likely work on this later this year (Q3?) which should also give extra time to make the app directory more stable

[gurkerl83]

Maybe something like like this can help to omit loading all translation files at ones, esp. the caching strategy is interesting here one variant of the server part. https://github.com/QuiiBz/next-international/blob/v2/packages/next-international/src/app/server/create-get-scoped-i18n.ts
The client part especially how translations are instructed to load is also interesting, also this requires no plugin.

Maybe this helps
Thx!

[amannn]

In next-international, getScopedI18n is about being able to provide a namespace, which is not supported by getI18n as far as I understand. Loading of messages upfront via createI18nServer/createI18nClient is the same for either API. That's at least my understanding.

next-intl handles both cases with one API via getTranslations(namespace?) and useTranslations(namespace?).

The current plan for this feature is to collect namespaces and keys with a compiler!

[gurkerl83]

I apologize for the misunderstanding in my earlier message about scoped translations - I should have read the documentation more thoroughly. Scoped translations help streamline code by reducing redundancy.

getI18n vs getScopedI18n

const t = getI18n();
console.log(t('pages.settings.published'));
console.log(t('pages.settings.title'));

const t = useScopedI18n('pages.settings');
console.log(t('published'))
console.log(t('title'));

About the last sentence...

The current plan for this feature is to collect namespaces and keys with a compiler!

In which context, the issue about tree shaking or how next-intl currently handles internals?

Thx!

[amannn]

No worries! Yep, I agree that scoping translations is helpful. next-intl is heavily based on that too! 🙂

[stylehuan]

Is there any progress?

[stramel]

@amannn Will this be covered in the v4 release? #1412

[amannn]

@stramel Not yet, no. I was generally a bit busy with other tasks recently, but I'm planning to pick up this topic again early next year to hopefully introduce this in the v4 range.

The current state is that there's a promising proof of concept (see #1 (comment)) that seems to show that this can be done. I'd like to explore this topic in combination with #962 since these topics are related.

I'll keep this issue updated once I have more. Thanks everyone for showing so much interest in this feature, I absolutely know this is an important feature for many of you!