Turbopack can't locate serverExternalPackages if installed via pnpm and are child dependencies of other installed modules #68805
Comments
jmikrut
changed the title
|
@jmikrut Not running into this error myself |
|
Hey @samcx you need to load up the admin UI (the spot that uses the dependencies in question). I forgot to add that to the reproduction instructions - good catch. Updated! If you're still not able to repro, I also noticed that you're on |
|
I don't get that error (I'm on pnpm 9.7.0) The |
|
@jmikrut It seems error out because I don't have the correct keys. Where do I should I go create my keys (ideally, a |
|
Ah apologies for that, let me update with a simpler repro that doesn't require keys. 1 sec |
|
@samcx I updated with a proper repro that simplifies and does not require environment variables. Should just need to clone the newly linked repro above and run it with turbo to see it fail. |
|
Hey @samcx I created a very, very minimal reproduction of this issue here: https://github.com/AlessioGr/very-minimal-turbo-bug-repro It's a blank, fresh Next.js project that includes a small, local package in the The import { drizzle } from 'drizzle-orm/libsql';
import { createClient } from '@libsql/client';
export async function test(dbURL) {
const client = createClient({ url: dbURL });
const db = drizzle(client);
console.log('db', db);
}It has the following dependencies: Calling Within the Next.js app/page.tsx we're using that package like this: import { test } from 'mydep';
export default function Home() {
test('file:./test-sqlite.db')
return (
<p>
test
</p>
);
}Super simple! External package uses drizzle-orm and @libsql/client to connect to sqlite - that external package is then used within the page.tsx. When running When running I shouldn't have to install But Turbopack currently does not respect this. It would be a nightmare for package authors to force their users to install their packages' own dependencies |
|
Hmm, it seems to not just be that package. I was able to confirm! I will be sharing this with the |
|
Good news, it was already reported internally—we are taking a look! |
|
Fantastic - thank you all! |
|
Did something happen in canary 114? |
|
@khuezy what version were you using before? |
|
(Responding to #68805 (comment)) Due to various implementation detail intricacies in pnpm, that does indeed work because that external package is available even though it wasn't actually declared anywhere as a dependency to pnpm. To demonstrate how the old (Webpack) behavior could break your app, I modified your reproduction: https://github.com/mischnic/turbopack-externals-repro (commit 571437bb) to trigger a just-as severe or even worse failure mode where the external dependency version might change after building.
So now mydep has actually imported With growing project size, this becomes more likely because there are simply more dependencies and possibilities for hoisting to not do what you want. Turbopack fails and tells you what is going wrong: (And in the next canary, will also include the offending version numbers: #68871) This is why I think that the Turbopack error message is helpful to prevent these kind of (hard to diagnose) problems. To maybe explain some of the behavior you're seeing with older versions: Before canary 84, Turbopack didn't externalize dependencies inside node_modules (so if libsql should be external, that would only have worked in user code and not when importing libsql in some other package), now it behaves like Webpack. |
|
Hey @mischnic I see why a version mismatch is an issue and why it makes sense to throw an error there - we do something similar in our project for critical dependencies. However, I think this might be a separate issue than what we're experiencing here. I see how it makes sense in your modified repro since there are 2 conflicting versions of the dependency installed, but in my repro, there is only a single version of the drizzle-orm / libsql dependency installed. The external package declares it, and the consuming Next.js project does not. The underlying error is also different. Yours mentions
which totally makes sense. However, our error is
No dependency version mismatch |
|
@samcx it may be a mismatch version the others have stated. I had an older version of https://github.com/vercel/next.js/releases/tag/v15.0.0-canary.113 |
|
@mischnic your notes make sense, and we understand your reasoning, but we flatly do not think that the current Turbopack functionality is acceptable. (AlessioGr is on my team). We would rather the onus of mismatching versions be on the developer, so they can be responsible for the risk you've described rather than Turbopack becoming non-functional even in cases where versions match. It is strange that there are inconsistencies with package managers (dev needs to now think about which package manager they use in order to use Turbopack). How does your implementation of Webpack currently accomplish this? This all works in Webpack correctly. The Node ecosystem would Example, if we install DependencyA which installs and relies on DependencyB, then DependencyA would As Turbopack works now, a I really think that if we have to tell people to install all sub-dependencies of our packages in their repo, they will simply be inclined to continue using Webpack rather than Turbopack. In any case, thank you for your continued insight! We just want to voice our concern and ensure that we know how to proceed here (if we do indeed have to tell end-users of Payload to install our sub-dependencies manually or not). |
|
+1 Experiencing the same error with Prisma by following this guide |
Webpack simply doesn't have these error messages, the output is/would be the same.
It's not really a question of Webpack vs Turbopack. It actually depends on the package manager your end-users use: I've updated https://github.com/mischnic/turbopack-externals-repro to use npm instead of pnpm, and it actually fails with This is precisely the situation Turbopack also shown the error, only it's more actionable because you know why it's happening (an external), and the error is shown deterministically at build time. |
|
Sorry for delay in response today - I have not had the chance to go deep here until just now. Keeping Turbopack working with Payload as it changes is a top priority of ours and I'll try to rapid-fire the best I can.
So in addition to using NPM, your repro actually adds the packages to Webpack externals, which we are in control of in userland and I would be happy to concede that yes, that could break things. But we're not marking the offending packages as externals in Webpack ( If you remove those lines that you added to your repro from I don't know the difference between marking something as a Webpack external vs. what Things used to work great with Turbo prior to canary 84. The fact is that we had no runtime errors prior, but now we do (only with Turbopack). And I understand that Turbo now properly externalizes You are all obviously the experts here (I am out of my league). I don't have strong feelings regarding which mechanism is used to resolve this. But in the event there are no version conflicts, I REALLY don't want to force my end-users to define dependencies in their own package.json which we already ship for them. So I would be very grateful for a way around that, whatever it is. |
timneutkens
added
the
linear: turbopack
That was already in the original repro from @AlessioGr, I just had to use drizzle-orm instead because I wasn't able to trigger a version mismatch with
I didn't know this either until today, but Webpack has the counter-intuitive behavior that if an import resolves to a difference version from the project root, Webpack just ignores that next.js/packages/next/src/build/handle-externals.ts Lines 102 to 105 in 0c74267
Have you got some example from that time I could run? |
|
Do you know/remember why you are externalizing |
Woah, strange. So that's the discrepancy then.
Yep, we can hook you up. So, SQLite is a brand new database adapter that we support and it looks like it's currently incompatible with Turbopack regardless, so to properly test here we'd want to swap out that adapter and use one that we know works (like Postgres / MongoDB). But those will require you to have a database connection to test with. Do you have a preference for which adapter you'd like us to remove SQLite in favor of? Whichever's easier for you to supply a connection string. Then we'll make a repro.
We're externalizing As for pino and pino-pretty, at one point we had to externalize them because they would break Webpack / Turbopack if bundled. I would love to not externalize them! I can do some checking to see if that is still the case. I remember it was super tough to get them to bundle properly and that was the fix, but maybe that's changed now. Will get back to you on that. Give us a bit here and I will get back to you. Thanks for the recon! |
|
OK - we have a repo using Postgres set up that demonstrates the issue. https://github.com/jmikrut/nextjs-68805-pg For clarity, I've added our entire But Turbopack still errors on the Does this help? |
|
Great, I'll take a look!
You mentioned this in the past as well, can you provide more details? Is it that error where it fails to load the (correct) native binding when including |
|
Yep - we haven't troubleshooted that It would be great to be able to get to the bottom of that one too! Different issue though. I totally am OK with dodging that one for now. Might even be something that |
The latest Turbopack in canary 120 compiles that just fine when not making Turbopack in canary 80 also bundled it, because of the fix for transitive externals that only happened in canary 83. So we should be able to just remove that one. (I did that test by removing that package from
The problem is this very dynamic require, it might be possible to do more code analysis in Turbopack to fix this. But Webpack also throws a build error when not externalizing that. |
|
Oh nice, so Yes, I figured the |
I'm not sure if I hit all code paths at runtime, but apparently it works.
They also appear to bundle just fine in your example https://github.com/jmikrut/nextjs-68805-pg
I found these |
Have the same issue as this, I had to de-activate turbopack mode for my project for now |
|
Happens on Shiki as well |
|
A temporary workaround is to use For example, for me I was getting the error for the following packages; public-hoist-pattern[]=*keyv*
public-hoist-pattern[]=*rimraf*Then just run pnpm install again. |
This comment has been minimized.
This comment has been minimized.
|
Same issue here, it's probably related to
|
|
Encountered the same issue on |
|
Same issue here. Using Payload beta 116. ▲ Next.js 15.0.0-rc.1 (turbo)
✓ Starting... ./node_modules/.pnpm/pino@9.3.1/node_modules/pino/pino.js ./node_modules/.pnpm/pino-pretty@11.2.1/node_modules/pino-pretty/index.js |
|
For me, after upgrading drizzle-orm from 0.33.0 to 0.35.2 and @libsql/client from 0.10.0 to 0.14.0 results into: |
|
A general update here: we made this a warning (also in the just released Next.js 15) instead of a hard error. @GustavoOS Could you try Next.js 15 and if you still get a |
|
I have moved to Bun because of this issue, and there have been no issues so far. It's also affecting other libraries, not just Drizzle. |
@mischnic the issue still occurs with Next.js v15 that was released today. In my case, it was because of the Better Auth implementation and showing the following: Package oslo can't be external
The request oslo matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install oslo from the project directory.Installing |
It seems to work on Next 15 with Drizzle |
|
So is the solution (for now) to move off |
This comment has been minimized.
This comment has been minimized.
|
Hello, Any updates for this issue ? I've already this error... Thanks you
|
|
Getting the same just trying to use pino from an internal common package in pnpm monorepo |
|
I was getting the following warning when running turbo with sentry installed Then I installed This at least hid the warnings for me |
|
Hey Next.js team - thank you for making these warnings instead of full-on errors. That is a good step. Everything works and is fully functional at this point but WOAH there are lots of warnings being logged. Looks like the warnings log on every request. Maybe you could consider adding a flag to silence these specific warnings in the Next.js config? That would be huge and with that I'd be all set on this issue. |
### What? Disables these annoying warnings when running `pnpm dev --turbo` <img width="656" alt="image" src="https://github.com/user-attachments/assets/7d0a2990-b5fd-4f5d-a025-665e8e3a7880"> vercel/next.js#68805 ### How? Patches `console.warn` in `withPayload.js` Can be disabled with `PAYLOAD_PATCH_TURBOPACK_WARNINGS=false` env variable
|
@jmikrut Can you share a screenshot of these warnings? I would usually lean on fixing the warning instead of outright silencing it. Silencing a warning may lead to the warning never getting fixed. |
|
I don't understand what is there for users to fix. Is this suggesting that I install this Example 2: As you can guess from the path, this dependency comes from the build folder of another package in a monorepo. But the main project only imports the auth package Finally, here is an example of my terminal log, just to give you a feel of how annoying this is in terms of DX: |
and others
Link to the code that reproduces this issue
https://github.com/jmikrut/nextjs-68805
To Reproduce
pnpm dev --turboCurrent vs. Expected behavior
Turbopack fails to compile, and the terminal blows up with errors such as the following:
If you install dependencies with Yarn, or with
npm i --legacy-peer-deps, the dependencies are located correctly with Turbopack.Note that all dependencies which error are indeed installed as dependencies of installed packages. Webpack works, but Turbopack does not.
Provide environment information
Operating System: Platform: darwin Arch: arm64 Version: Darwin Kernel Version 24.0.0: Wed Jul 31 21:51:10 PDT 2024; root:xnu-11215.0.199.501.2~1/RELEASE_ARM64_T6000 Available memory (MB): 32768 Available CPU cores: 10 Binaries: Node: 20.11.1 npm: 10.2.4 Yarn: 1.16.0 pnpm: 8.15.7 Relevant Packages: next: 15.0.0-canary.111 // Latest available version is detected (15.0.0-canary.111). eslint-config-next: 15.0.0-canary.104 react: 19.0.0-rc-06d0b89e-20240801 react-dom: 19.0.0-rc-06d0b89e-20240801 typescript: 5.5.4 Next.js Config: output: N/AWhich area(s) are affected? (Select all that apply)
Turbopack
Which stage(s) are affected? (Select all that apply)
next dev (local)
Additional context
This appears to have broken in
next@15.0.0-canary.84. Worked before that.The text was updated successfully, but these errors were encountered: