next-transpile-modules and transpilePackages: how do we close the gap?

[martpie]

Hi 👋

maintainer of next-transpile-modules here.

@timneutkens, mentioned the new transpilePackages option that landed with Next.js 13, so I thought it may be finally the time to deprecate my plugin, that is widely used by the community, with almost 25% of Next.js setups using it (800K weekly downloads, 3385K for next).

Long story short: transpilePackages is a very welcome addition, but is not fully ready for prime time yet, and next-transpile-modules covers more usecases. But the usecases gap may not make sense to be supported on Next.js.

I'll detail things a bit more below.

There is a main question to be answered: how do we close gap? And more importantly, do we even close the gap?

---

next-transpile-modules uses a 4 dimensions (2 primary, and 2 secondary) matrix to test various usecases:

1st dimension: Use cases

• test-local-module (import and use local ES module)
• test-npm-module (import and use npm ES module)
• test-local-typescript-module (import a local module written in TypeScript)
• test-css-module (import local component importing a css module)
• test-scss-module (import local component importing a scss module)
• test-global-ccss (import local component import css to be injected globally)
• test-global-scss (import local component import scss to be injected globally)

2nd dimension: Setup

• npm
• yarn workspaces
• yarn workspaces with local symlinks
• pnpm
• swc

3rd dimension: Node.js - secondary

• Node.js 14
• Node.js 16

4th dimension: Operating system - secondary

• darwin
• windows
• linux

I'll primary talk about the 1st dimension, as the other ones are already covered/supported by Next.js in general.

So, let's get back to our list of use cases, and let's see what's compatible with the official transpilePackages:

last update: Nov 19th, Next.js 13.0.4 (🎉)

	npm's file:/	yarn-workspaces	yarn with symlinks	pnpm
test-local-module	✅	✅	✅	✅
test-npm-module	✅	✅	✅	✅
test-local-typescript-module	❌	✅	✅	✅
test-css-module	❌	✅	✅	✅
test-scss-module	❌	✅	✅	✅
test-global-ccss	❌	✅	✅	✅
test-global-scss	❌	✅	✅	✅

So, if I have to simplify:

• ✅ import ES modules
• ❌ import TS modules
• import modules making use of CSS features (global or local css)

Now, the main question: does the Next.js team plan on covering these usecases at some point? And if yes, do we have an vague idea of a potential release date? (are we talking weeks or years?)

Depending on Next.js team vision, it will heavily influence what's next for next-transpile-modules, let it be a deprecation, a scope reduction, or a better integration with transpilePackages.

As always, I remain at the Next.js team disposal if you would like to learn more about why those usecases exist (I've built my fair share of Next.js apps in the past 5 years), and how we can help the community to get rid of next-transpile-modules once and for all. :]

cheers 🚀

[timneutkens]

Thanks for the thorough check @martpie! @shuding is already working on adding these to get feature parity 👍

[martpie]

Updated the post with a more complete compatibility table with Next.js 13.0.4. Fantastic work from you peeps, congrats!

[timneutkens]

Thanks! Will have Shu take a look at the last few 🙏

[shuding]

#43203 should fix the test-local-typescript-module case, after that I'll check the remaining ones. Thanks for the pointers @martpie! :)

[cereallarceny]

On an emotional note, and not like I ever expected any less, but what an incredibly professional and mature way to approach this topic @martpie. Good on you for putting the greater community before the individual ego. Looking forward to seeing more updates on the topic here! 😄

[astateful]

Hello everyone! I am a super happy user of next-transpile-modules and it is great to see this functionality being supported natively by nextjs. Since I do not see styled-components mentioned in the 1st dimension use case, I am curious as to what the status of support for transpiling styled-components is.

Assuming the new transpilePackages option is intended to support styled-components (since next-transpile-modules does) there is an issue that I have reported, available here: https://github.com/vercel/next.js/issues/46422. Transpiling styled-components via transpilePackages option does not seem to be working. If there is an alternative way/method/technology to achieve this, it would be great to learn about it and adapt my methods, else it is the only issue preventing me from using the transpilePackages option.

[avo-SS]

Any updates on this?

[rune-hansen]

@martpie I see that you have marked your fantastic "next-transpile-modules" as deprecated with a comment that all features are now covered by next.js itself. As can bee seen by @astateful comments above this is not entirely true.
Actually the missing piece is a right pain for our company since we are not allowed to use deprecated packages and next.js does not have the functionality yet.
My cheeky question is if it is possible for you to un-deprecate the next-transpile-modules package so we and I am sure others can make use of it again until next.js is up to speed.

[martpie]

Hey @rune-hansen 👋

we are not allowed to use deprecated packages

That's a shame :( next-transpile-modules is not "deprecated" per-se, but the project is "done".

From what I understand, the native transpilePackages is missing support for esoteric setups using symlinks, and we should:

• try to make the Next.js team aware of it (poke @timneutkens @shuding)
• if that's urgent, you should try submit a patch to Next.js. The team is very nice and will probably help you there. The discord might be a good place to start.

My more personal and opinionated take on this is you should absolutely move your setup away from symlinks as much as possible, they're a pain. Use monorepos if you can, and workspaces. I understand that's easy-said, but I cannot stress enough how simpler this setup is at scale and you will save yourself some white hairs.

If that can temporarily help your setup, I can un-deprecate it on npm, but the repository will probably still stay archived, or at least won't receive any update from me.

edit: I have removed the npm deprecation, in case that can help your work. But my previous sentence still applies :D

[astateful]

Hello martpie! Thank you so much for your great insight. My repository is currently using a monorepo setup via yarn workspaces. When I switched away from next-transpile-modules package to the native transpilePackages option, the transpilation of styled-components stopped working.

As I understand yarn workspaces, there are a set of symlinks created in the root node_modules folder for all packages in the defined workspaces. These packages can then be imported like any other NPM-hosted package within the application. These are the only symlinks I was aware of that are being used in this setup. And as far as I know, these symlinks are managed via Yarn workspaces. Are you suggesting it is these symlinks we move away from?

Would it be possible for you to elaborate more on the symlink support that is missing from NextJS and how it relates specifically to this issue? Would be great to understand the problem better. Are symlinks somehow being "mishandled" by transpilePackages option?

[airtonix]

@martpie it would be good if you could elaborate on this in some way (links, additional comment in OP) about this, since it's also my understanding that on linux that this is how yarn,npm and pnpm work in a monorepo scenario.

[airtonix]

@astateful btw it might be worth knowing that nxdev provides tooling around monorepos and managing nextjs apps in such monorepos: https://github.com/nrwl/nx/blob/master/packages/next/plugins/with-nx.ts#L201-L218

[iDVB]

We seem to only be able to get transpilePackages + monoRepo + styled-components if our custom ui lib uses "jsx" files. Otherwise we keep getting...

⨯ TypeError: styled_components__WEBPACK_IMPORTED_MODULE_3__ is not a function
> 10 | const StyledTypography = styled(Typography)