[Question] How does the topological pnpm run strategy work

[michaelwarren1106]

I have a monorepo that I’m testing switching to pnpm, and my packages have several cross-pkg dependencies.

For example, package/a depends on core and package/a won’t build correctly unless core has been built first.

I’ve tried to run pnpm -r build with the understanding that pnpm will look at the dependencies of a package and run the same script on dependencies first. In my example, this would mean that core’s build script gets run first then the build script for package/a.

But i don’t see this happening, and the topological order of script runs isn’t very well documented.

Another related question is about symlinking those same dependencies. when i run pnpm i at the root and i’m using linkDirectory:true to symlinking each of my packages dist folders, it seems that pnpm won’t create the symlink unless the folder exists. Is there an option to create the symlink anyway since the dist folder will exist later?

[zkochan]

Yes, it should run the scripts in topological order, unless you use the --no-sort option. Are your local deps added using the workspace protocol?

Another related question is about symlinking those same dependencies. when i run pnpm i at the root and i’m using linkDirectory:true to symlinking each of my packages dist folders, it seems that pnpm won’t create the symlink unless the folder exists. Is there an option to create the symlink anyway since the dist folder will exist later?

I guess it should work. If it doesn't work, it is probably a bug.

[michaelwarren1106]

not yet. my dependencies are just using package name and version as normal, not workspace:<version>

[michaelwarren1106]

is the workspace protocol required for topological ordering?

[zkochan]

The package.json in the dist folder may contain the workspace protocol entries. pnpm will replace them when packing the package.

[michaelwarren1106]

ok cool! so i just need to copy it in there before the changeset commands?

what about symlinking before the dist folders are created? is there a way to tell pnpm to create the symlink to dist even if the folder doesn’t exist so that when builds are run later the symlinks are already there?

i think the symlinking to non-existent linked directories is my last hurdle

[zkochan]

so i just need to copy it in there before the changeset commands?

not before. After

is there a way to tell pnpm to create the symlink to dist even if the folder doesn’t exist so that when builds are run later the symlinks are already there?

it should already work like that. If it doesn't, then it is a bug

[michaelwarren1106]

@zkochan I just tested in a clean repo (no node_modules or dist folders)

All my packages have:

publishConfig: {
  directory: 'dist',
  linkDirectory: true
}

when i run pnpm i, i get warnings like:

WARN  Local dependency not found at <path>/base/dist
 WARN  Local dependency not found at <path>/tokens/dist
 WARN  Local dependency not found at<path>/style/dist

i then manually went to my base package and built it, and checked another package that depends on base, and the symlink was NOT created and there was no linked package in node_modules

so I think we've got a bug. Is this enough info to test with, or do you want me to put together a full repro?

[michaelwarren1106]

i tested with and without the workspace protocol and neither worked

[mkstix6]

Could we have a little documentation on how the "topological" algorithm works?
And if there is any setup or prerequisites?

It would be nice to read something that confirms I understand it correctly.
More specifically I'd like to feel confident that topological sorting does what I think for a command such as…
pnpm -F packageA -F packageB -F packageC build
…where:

1. the order packages build matters
2. dependencies listed in package.json
3. build scripts for each package listed in package.json scripts

Using the documentation's search for "topological" I do see references to it in terms of "use this flag to turn it off if you want" which is a bit of an understanding-by-deduction exercise.

Thanks very much ✨. Really great work on pnpm.

[shanefontaine]

Looking at the code, I do not believe the lifecycle scripts respect by the topological sorting algorithm. I'll walk through the relevant code (as I understand it) below. @zkochan I'd greatly appreciate it if you could confirm the accuracy of these observations or correct any misinterpretations.

1. Installing workspace packages with pnpm install runs installDeps().
2. This function ignores all lifecycle scripts while installing deps. It then calls rebuildProjects() with the intention of now installing the lifecycle scripts.
3. This function calls runLifecycleHooksConcurrently() which appears to run the lifecycle scripts for each package in parallel (but does run each individual package's scripts serially, as you would expect).

Happy to open a PR for code or docs about this to help future readers, but want to confirm that my understanding aligns with the intended behavior of pnpm. Thank you for your work 🙏

I recognize this is an unsolvable problem but hope to get some clarity.