Typings directory when in-lining project references

[lavcraft]

Troubleshooting

My reproducible example - https://github.com/lavcraft/rollup-typescript-project-references-example-problems

You need

1. clone
2. npm run ci

Follow to next instructions

1. Does tsc have the same output? If so, please explain why this is incorrect behavior

Run npm run -w @project/sdk build - tsc variant works fine and generate d.ts files into correct directories

1. Does your Rollup plugin order match this plugin's compatibility? If not, please elaborate

I guess order is correct:

        nodeResolve(),
        typescript2()

1. Can you create a minimal example that reproduces this behavior? Preferably, use this environment for your reproduction

My example - https://github.com/lavcraft/rollup-typescript-project-references-example-problems
Follow to README or read next lines:

Two commands for feel difference between tsc and rollup in my case:

TSC baseline

1. npm run -w @project/sdk build - tsc variant
2. And see d.ts files in dirs

tree packages/sdk/build
packages/sdk/build
├── actions
│   ├── create.d.ts
│   ├── create.d.ts.map
│   └── create.js
├── models
│   ├── user.d.ts
│   ├── user.d.ts.map
│   └── user.js
└── tsconfig.tsbuildinfo

All d.ts files existed in right location (except d.ts from project from project reference, but it is ok for tsc i guess)

Rollup variant

1. rm -rf packages/sdk/build && npm run -w @project/sdk rollup
2. And see d.ts files in dirs

tree packages/sdk/build
packages/sdk/build
├── actions
│   ├── create.js
│   └── create.js.map
├── models
│   ├── user.js
│   └── user.js.map
├── sdk      <------------------------ THIS DIR SUSPICIOUS
│   └── src
│       ├── actions
│       │   ├── create.d.ts
│       │   └── create.d.ts.map
│       └── models
│           ├── user.d.ts
│           └── user.d.ts.map
└── shared <------------------------ THIS DIR SUSPICIOUS
    └── src
        └── test
            ├── util.d.ts
            └── util.d.ts.map

Created to excess directories with typings - sdk and shared

Please help me to explain right behaviour or workaround. Because my next step is apply dts plugin for obtain valid d.ts files without internal paths. But some difficulties happened when all this d.ts located in strange directories :)

Environment

Versions

  System:
    OS: macOS 13.3.1
    CPU: (8) arm64 Apple M1 Pro
    Memory: 52.64 MB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.16.0 - ~/.nvm/versions/node/v18.16.0/bin/node
    npm: 9.5.1 - ~/.nvm/versions/node/v18.16.0/bin/npm
    pnpm: 8.6.1 - /opt/homebrew/bin/pnpm
  npmPackages:
    typescript: ^5.1.5 => 5.1.5

rollup.config.js

:

import commonjs from '@rollup/plugin-commonjs';
import {nodeResolve} from '@rollup/plugin-node-resolve';
import typescript2 from 'rollup-plugin-typescript2';
import fs from 'fs';
import path from 'path';
import * as url from 'url';

const __dirname = url.fileURLToPath(new URL('.', import.meta.url));
/**
 * @type {import('rollup').RollupOptions}
 */
const config = {
    input: {
        ...files('actions'),
        ...files('models'),
    },
    output: {
        sourcemap: true,
        dir: 'build',
        preserveModules: false,
        format: 'esm'
    },
    plugins: [
        nodeResolve(),
        typescript2({
            // build: true,
            // useTsconfigDeclarationDir: true,
            // declarationDir: './types',
        }),
        // commonjs(),
    ],
    external: [
        'undici',
        'querystring',

        'mobx',
        'mobx-react-lite',
        'mobx-utils',
        'react',
    ],
    watch: {
        chokidar: true,
        exclude: ['build/**', '**/*.d.ts', '**/*.js']
    }
};

function files(dir = '.', exclude = ['__tests__']) {
    return fs
        .readdirSync(path.resolve(__dirname, `src/${dir}`))
        .reduce((prev, current) => {
            if (exclude.includes(current)) return prev;
            const targetName = current.replace('.ts', '');
            prev[`${dir}/${targetName}`] = `./src/${dir}/${current}`;
            return prev;
        }, {});
}
export default [config];

[agilgur5]

Oh there are indeed some complications in your case. I see you have configured TS paths, and that it goes up two directories. That would indeed set your rootDir to be packages.

Unfortunately, aliases are not truly supported by TS. #201 has pretty significant details and several workarounds to support aliases. IIRC, there are some related issues on monorepos specifically that I might be able to pull up as well (#237 is one)

Using one of those workarounds can probably fix your issue. Also not sure if @shared is meant to be compiled in or meant as a separate external package reference when published.

[lavcraft]

Thanks for this workarounds:
#237
#201

But in my opinion my example highlight another kind of problem. I try to process resulting declarations, but it's hard, because located in strange directories (also located strange dir when i set declarationDir, but also copied into declaration dir with <workspace_dir>/src prefix)

I guess it's not reliable behaviour (i may be wrong). I am trying to find a consistent solution to avoid problems in the future.

In my example with rollup, shared workspace sources are inlined into the resulting sdk perfectly. But types have some problems, because previously when i have used @rollup/plugin-typescript, I process resulting d.ts files via dts plugin with respectExternal prop and inline all my shared d.ts into sdk project and replace paths in declarations. But with typescript2 this workaround doesn't work

[lavcraft]

@agilgur5 could you help me with one more advice for avoiding excess .d.ts files generation in <workspace_dir_name/src/**/*.d.ts dirs? Workarounds from previous message related to other issues or not working in this case i guess

[agilgur5]

goal? in-lining?

In my example with rollup, shared workspace sources are inlined into the resulting sdk perfectly.
inline all my shared d.ts

Ah ok, so you are trying to in-line the JS without in-lining the types? Or in-line both?
In-lining both is what rpt2 is currently doing in your repro example.

In your repro example, you might notice that npm run -w @project/sdk build, the tsc variant, just treats @shared as an external:

// create.js
import { test } from "@shared/test/util";
export function create(user) {
    console.log(`user = ${user}`);
    test(() => { });
}

tsc will not in-line @shared and does not produce declarations for @shared, as that's a separate project.

With Rollup, you can mark @shared as an external to not in-line it.
Unfortunately, however, rpt2 is still producing declarations for @shared. I was able to get it to not produce declarations for @shared when I added it to rpt2's exclude, however it still outputs sdk's declarations in build/sdk/src/.

root cause analysis

I spent a good amount of time trying to debug this and unfortunately hit a bit of a wall 😕

This seems to be because rpt2's internal TS LanguageService is still interpreting your rootDir as packages instead of sdk/src.
I was able to confirm that by setting rootDir: 'src/' and then tracing the error with verbosity: 3. rpt2 is getting an error during semantic diagnostics on create.ts, which it leaves entirely up to the TS LanguageService.

So for some reason the LanguageService is interpreting your project reference differently from tsc. Unfortunately, the LanguageService API is barely documented, so folks writing TS compiler integrations (like rpt2) have to effectively "reverse engineer" some of the behavior. This seems to be one of those instances where we have to do that, as it's unclear why it's acting differently right now when given the same tsconfig.
I dug into the TS Compiler source code a bit but this area of it is really dense with variables referencing variables referencing variables in a very long and deep chain.

As a note, it had this behavior even when I removed the aliases and just manually referenced the other project, so this does not seem to be due to integration with TS paths.

EDIT: It actually was due to the paths aliases 😕 . See below comment.

temporary conclusion

Given the significant analysis / reverse-engineering and debugging this requires, I don't think this will be solved soon (unless someone happens to make a really good guess). As such, I would recommend using a workaround for now

[agilgur5]

OH. I got it to work.

As a note, it had this behavior even when I removed the aliases and just manually referenced the other project, so this does not seem to be due to integration with TS paths.

It seems like it may indeed be paths related. Specifically, I changed your paths to use the output directory, shared/build/*, instead of shared/src/*:

     "paths": {
       "@shared/*": [
-         "../../shared/src/*"
+         "../../shared/build/*"
       ]
     }

and then it worked with no other changes needed 😮
it output .js and .d.ts to build/actions and build/models with no extra build/sdk/src directory for the declarations.

adding /@shared/ to external will get rid of the Rollup warning as well.

EDIT: sent a PR with fixes: lavcraft/rollup-typescript-project-references-example-problems#1

[lavcraft]

@agilgur5 yes, main purpose is inlining both - js and .d.ts. But secondary - simply build and make it more reliable (across versions and updates). That mean, i wan't to use ts project reference for use rollup dev mode and auto build dependencies and caching. If i change paths.@shared from ../../shared/src/* to ../../shared/build/* i miss this ability

I believe - separate code into sub projects — provide more understandable project structure without limits

[lavcraft]

In my opinion, current behaviour in my example contain only one thing, that should be fixed for purposes. And how it can be achieved in manual way:

1. npm run -w @project/sdk rollup
2. Copy all d.ts from build/sdk/src and build/shared/src to appropriate builddir.
3. Delete build/sdk and build/shared
4. Run rollup second times on resulted .d.ts for inline @shared import

[lavcraft]

This problem looks like rpt2 plugin doesn't add sources from referenced projects into rootDirs set, however compile and build related files

[agilgur5]

That mean, i wan't to use ts project reference for use rollup dev mode and auto build dependencies and caching. If i change paths.@shared from ../../shared/src/* to ../../shared/build/* i miss this ability

So I responded in lavcraft/rollup-typescript-project-references-example-problems#1 (comment), but this was working fine for me.

use rollup dev mode

Not sure what you meant by this, but Rollup can't really process project references the same way tsc can. That requires starting two Rollup processes and having two rollup.config.js. rpt2, as a Rollup plugin, cannot do that itself -- that would be out-of-scope.

yes, main purpose is inlining both - js and .d.ts.

So in this case, the current behavior is correct. The common rootDir between shared and sdk is one directory up, in packages. So TS is calculating the common rootDir correctly and outputting declarations based on that rootDir, giving you the sdk/src/ and shared/src directories for declarations. I wrote this in lavcraft/rollup-typescript-project-references-example-problems#1 (comment) as well.

This problem looks like rpt2 plugin doesn't add sources from referenced projects into rootDirs set

I'm not sure what you mean by this. As far as I know, TS itself does not add project references into rootDirs either? Those are separate configurations.
Either way, rpt2 passes your tsconfig directly to TS (with a few changes for Rollup compatibility, as stated in the README).

In my opinion, current behaviour in my example contain only one thing, that should be fixed for purposes.

Sorry, what is the "one" problem that remains? It seems like everything is working as intended in your list

[lavcraft]

So I responded in lavcraft/rollup-typescript-project-references-example-problems#1 (comment), but this was working fine for me.

It works only if shared project builded. In this way, i need to run independent process for building shared project, and then — rebuild sdk project when build/*js was changed. I want to simplify project build with ts project reference, but this thing with src -> build in paths nullifies efforts

Not sure what you meant by this, but Rollup can't really process project references the same way tsc can. That requires starting two Rollup processes and having two rollup.config.js. rpt2, as a Rollup plugin, cannot do that itself -- that would be out-of-scope.

Sure. It's main goal — avoid complex build with multiple process, and replace it with project references. I use comparison with tsc only for d.ts generating, because:

1. tsc generate valid d.ts files(only without @shared inlining)
2. rollup inline @shared in js code, but generate strange d.ts structure. I expect, rollup generating similar d.ts with inline @shared — but in this way i have a problem with strange directory structure only for types and can't understand how to correctly run dts() plugin on this types for "inline" @shared imports

Sorry, what is the "one" problem that remains? It seems like everything is working as intended in your list

Sorry, i mean problem with types :) And if i try to fix it with your workaround, unfortunately, nullifies profit from typescript project reference and need to run two rollup process for each project and manage it manually (because in paths i must reference into build dir instead of src)

[lavcraft]

Try to a new way:

I can adopt to new types structure with next package.json types:

  "typesVersions": {
    "*": {
      "*": [
        "build/sdk/src/*",
        "build/shared/src/*",
        "build/sdk/src/*/index.d.ts",
        "build/shared/src/*/index.d.ts"
      ]
    }
  }

@agilgur5 could you advice, how process types in sdk/src and shared/src directories for replace @shared alias in sdk/src/**/*.d.ts?

Found this solution — #201 (comment)
But i dont't understand how use this tscAlias function. My question:

1. I need to create separate config with d.ts inputs from sdk/src and shared/src dirs instead of .ts and apply tscalias plugin?
2. Do i need rollup-plugin-dts plugin, or only tscAlias from comment above ?