Order script execution by npm dependencies

[JodiWarren]

When working in a monorepo where the packages have varied dependency chains, it is difficult to use WireIt to manage the building of all packages.

The user would effectively need to recreate the internal npm dependencies within the dependencies section of each build command, otherwise a build will result in many missing file dependency file errors.

To illustrate the problem I'm facing, say we had a monorepo with 26 packages, named A-Z. Let's also assume they build in alphabetical order.

• Package A has no internal dependencies
• Package B depends on Packages A and C.
• Package D depends on Packages A, B, and E.
• Package F depends on Packages Z, W, and T.

When the build starts, A builds fine, but the rest fail with errors. B cannot find C, D cannot find B (which failed) or E, and F cannot find any of it's dependencies.

---

The repo I'm currently evaluating with WireIt has approx 115 packages. Once the initial set of built dependencies exists, WireIt is very fast to work with, but that initial build step appears to be a blocker. The only way I can get it working right now is to use a different system to do the initial build.

Apologies if I've misunderstood either the way the project works or the scope of it!

[aomarks]

Yeah this is true, Wireit is different to tools like Lerna in that the entire build graph is completely explicit, and does not make any assumptions based on npm dependencies. If package B's build script needs to run package A's build script first, then package B must declare ../a:build in its Wireit dependencies.

The advantage of this is that you can be very fine-grained with your dependencies. Often times a specific script in a package doesn't depend on its dependencies being fully built; rather it only depends on a subset of that dependency's built outputs. Fine-grained dependencies like that can substantially improve parallelism.

Here's an example from the Lit monorepo, which takes advantage of this https://github.com/lit/lit/blob/main/packages/lit/package.json#L163. In this case, running TypeScript only requires its dependencies TypeScript scripts to run, but nothing else. And running Rollup doesn't actually require its dependencies Rollup scripts to run at all, so all Rollup scripts across packages can run in parallel.

The downside of this is that it can become very verbose, as you say, and is prone to getting out of sync as new dependencies are added.

---

One feature that should be coming soon is #23, which will let you write $WORKSPACES:foo meaning "run foo in all of this package's npm workspaces".

We have also considered a similar feature, but for the reverse direction, where you could write $WORKSPACE_DEPS:foo, meaning "run foo in all of my dependencies which are part of the same workspaces configuration".

So your project might look like this:

<root>/package.json

The root package has two workspaces. Running npm run build from the root builds both of them.

{
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "dependencies": [
        "$WORKSPACES:build",
      ]
    }
  },
  "workspaces": [
    "foo",
    "bar"
  ]
}

<root>/foo/package.json

The foo package depends on the bar package. It declares $WORKSPACE_DEPS:build in its build dependencies, which means bar:build runs before its own build. If we ever add another dependency, then the new dependency's build will run first too, without us needing to update the build dependencies directly.

{
  "name": "foo",
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "command": "build-me",
      "dependencies": [
        "$WORKSPACE_DEPS:build"
      ]
    }
  },
  "dependencies": {
    "bar": "^1.0.0"
  }
}

<root>/bar/package.json

The bar package doesn't depend on anything, but still declares $WORKSPACE_DEPS:build. For now that's a no-op since there are no dependencies, but if we added a dependency, its build would automatically become a build script dependency.

{
  "name": "bar",
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "command": "build-me",
      "dependencies": [
        "$WORKSPACE_DEPS:build",
      ]
    }
  }
}

Does something along those lines sound like it would address your concerns? You would still need to declare a dependency on each script where you need this behavior -- but only once.

[aomarks]

See also #119

[JodiWarren]

That might well solve it for us and I'll be really interested to see how well it works. So then WireIt would follow the dependency chain for each package that appears within the workspace, build out a dependency tree, and step through, building one by one? The cacheing should make that very fast for subsequent packages with similar dependencies.

I also think the scope for WireIt being different from (say) Lerna is totally reasonable. Lit has a reasonably small amount of packages, with a high degree of difference between them. An explicit and optimisable configuration is probably a great fit there.

Thank you so much for your response.

[aomarks]

That might well solve it for us and I'll be really interested to see how well it works. So then WireIt would follow the dependency chain for each package that appears within the workspace, build out a dependency tree, and step through, building one by one? The cacheing should make that very fast for subsequent packages with similar dependencies.

Yes, exactly!

I also think the scope for WireIt being different from (say) Lerna is totally reasonable. Lit has a reasonably small amount of packages, with a high degree of difference between them. An explicit and optimisable configuration is probably a great fit there.

Thank you so much for your response.

Just curious, are you using npm workspaces, or another tool like Lerna for managing your multi-package layout? And if you are not using npm workspaces, is npm workspaces something you have looked into/would consider switching to? (Asking because the feature described above is oriented around npm workspaces).

[JodiWarren]

Just curious, are you using npm workspaces, or another tool like Lerna for managing your multi-package layout?

We're currently using a combo of Lerna and Yarn, though we've almost finished moving to PNPM.

And if you are not using npm workspaces, is npm workspaces something you have looked into/would consider switching to?

We've looked into it, but it's not quite enough on it's own. However, we would strongly consider switching to it if WireIt met all of our other needs. The strong caching means that the local build speed improvement with a primed cache is vast, and we have high hopes for it improving our CI pipeline speed, especially with the Github Actions Cacheing integration.

[aomarks]

Closing in favor of tracking at #23