Proposal: Bundling TS module type definitions

[weswigham]

Relates to #3159, #4068, #2568, this branch, and this tool.

Goals

• Bundle declarations for TS projects to allow a library to be consumed with a single TS file, despite being many modules internally. Because of complicated internal module dependencies which would rather not be exposed to the consumer, this should flatten the exported module types as best as it can. (Ideally, completely.)

Proposal

When all of --module, --out, and --declarations are specified, the TS compiler should emit a single amalgamated .d.ts (alongside its single output js file). This .d.ts should be flattened compared to a concatenated .d.ts file. It should report collisions caused by scoping issues and import aliasing when flattening declarations into a single declare module. It should respect access modifiers when generating the DTS (only exporting things explicitly exported and types marked as public).

For example, given the following set of sources:
tsconfig.json:

{
  "compilerOptions": {
    "module": "commonjs",
    "declarations": true,
    "out": "mylib.js"
  }
}

a.ts:

export * from './b';
export * from './c';

b.ts:

export interface Foo {}

export class Bar {
    constructor() {
        console.log('');
    }

    do(): Foo { throw new Error('Not implemented.'); }
}

c.ts:

export class Baz {}

should create the .d.ts:
mylib.d.ts:

declare module "mylib" {
  export interface Foo {}

  export class Bar {
    constructor()
    do(): Foo
  }

  export class Baz {}
}

rather than:
mylib.d.ts:

declare module "mylib/a" {
  export * from "mylib/b";
  export * from "mylib/c";
}
declare module "mylib/b" {
  export interface Foo {}

  export class Bar {
    constructor()
    do(): Foo
  }
}
declare module "mylib/c" {
  export class Baz {}
}
declare module "mylib" {
  export * from "mylib/a";
}

and should report a semantic error when the following is done:
a.ts:

export * from './b';
export {Bar as Foo} from './b';
export * from './c';

as there will be multiple members named Foo (an interface and a class), since b.ts has exported interface Foo.

We should also have a semantic error when the following is changed from the original:
If we change c.ts:

export class Baz {}
export interface Foo {}

it should be an error in a.ts (since it's blanket exporting b and c), and the error should suggest to alias either c.ts's Foo or b.ts's Foo (or both) when reexporting them in a.

Internally, when flattening this aliasing becomes important - we need to track usages of the two original Foo's across the generated .d.ts and rename it to the alias created when it is reexported.

Unfortunately, to maintain ES6 compatability, while we can warn about this behavior with classes (since it's possible that a developer is unaware they're overriding a prior export), we still need to support it (or do we? The spec leads me to believe that attempting to export multiple members with the same name - even via export * - is an early syntax error). So it would be nice to have a compiler flag to mark the same kind of thing with classes (or namespaces) as an error, but also do the following by default:

We can do automatic name collision resolution, but that can result in unpredictable (or convention-based) public member names... but it must be done, I suppose. We could ignore reexported types since it's appropriate to do so in ES6 (following export * declarations can override previously defined members? maybe? system works this way at present - but that may just be system relying on transpiler implementers to maintain ES6 semantics), then we would need to create "shadowed" types at the appropriate level in the .d.ts - types whose original public access are overridden by later exports but whose types are still required to describe public function argument or return types. Naming these "shadowed" types could be difficult, but given that they only exist for type information and not for access information, a common (re)naming convention could be a desirable solution. Something akin to <typename>_n when n is the shadowed type number for that type, and renaming the shadowed type name to something else (<typename>__n and so on so long as the name still exists) if that collides with another exported type. Classes used in this way are rewritten to interfaces in the .d.ts, since a constructor function likely isn't accessible for a shadowed class (at least not at its generated exported type name).

Any feedback? There's a few alternatives to what I've suggested here, which is possibly the most conservative approach in terms of ability to error early but supporting ES6 semantics best. It's possible to silently ignore interface name collisions and rename those automatically as well, but since they're TS constructs and not ES6, I think it's okay to force more discipline in their usage.

Something I've been considering is also rewriting namespaces as interfaces in the generated .d.ts in this way to further flatten/unify the types, but this... might? not strictly be needed. I haven't come up with a strong case for it.

[weswigham]

I realize that I've forgotten to propose a way to define the entrypoint module (as is possible in #4434), and I suppose that warrants discussion, too.

The entrypoint is the location which (in the above example) TS considers the 'top level' for flattening types (in this example, a.ts). Ideally, this is the entrypoint to your library or application. All of the dependent files in a well-formed module-based TS project should usually be accessible from the entrypoint via either imports or triple-slash refs... however In TS's default mode of operation for the --module flag, TS ignores most relationships between included files and compiles them mostly separately, resulting in separate js and dts files for each. Like the proposal in #4434, dts bundling may also make sense to require a --bundleDefinitions [entrypoint] flag, like how bundling module sources could require the --bundle [entrypoint] flag.

On the other hand, rather than add a new compiler flag, we could consider all files specified as 'root' files as entrypoints when compiling with --module and output a definition file for each of them. (Meaning that all other files need to be accessed by those root files and compiled via that relationship.) Conceptually, we could do the same thing with #4434, rather than having the --bundle argument to specify an entrypoint. This does lose the meaning of the --outFile argument, however, since there are suddenly multiple output files (one for each root file) and none of them correlate to the --outFile parameter... So maybe it's best to not try to rely on traversing the dependencies ourselves and require an extra parameter specifying the entrypoint.

[NoelAbrahams]

@weswigham,

I'm generally in favour of this proposal. I think this feature would help to formalise the idea of a project in Visual Studio (i.e. types defined in namespaces compiled into a a single output file, that can then be referenced by other projects).

One of the problems with a single output file is that it makes life difficult when debugging in the browser. See #192 (comment). Ideally something can be worked out for that as well.

[weswigham]

#3159 implements most of this proposal, though it restricts it to ES6 or commonjs modules, uses slightly different terminology for tsc command line flags. and omits the type flattening aspect of this proposal. At the very least, it's probably an excellent starting point for talking about this.

[alexeagle]

some notes from doing this for Angular:

• should also report a semantic error when a symbol is re-exported to an entry point, but a dependent symbol is not (for example, declared supertypes, parameter types)
• we like to emit a namespace as well as a module, allowing users to use symbols from that global namespace without any import statements (eg. useful for ES5 users in VSCode to get intellisense).
• we might want to be able to use this typings bundle feature without having to use the runtime emit bundling feature from Proposal: Bundling TS modules for consumption and execution #4434 - for example if there are some errors producing a working emit because of private APIs, it would be nice if we could still use this to produce the public API doc.
• We like to preserve some comments so that tools can show inline doc when showing eg. a completion tooltip. Should probably explicitly mention comment handling in the proposal.

[alexeagle]

We can't drop our current .d.ts bundler without constructor visibility: #2341

[ffMathy]

What is the current state of this issue?

[mhegazy]

It is still in discussion. we have a PR for it, but there are still some open questions.

[ffMathy]

Can you point to this pull request?

[mhegazy]

I should clarify, the concatenation is already done in: #5090
The remaining piece is the flatting, here is the PR: #5332

[PavelPZ]

I think that d.ts bundle works very nice, see angular#5796.

Thanks a lot for it.

[heruan]

I jumped to/from many issues regarding this "theme". Is this the right one where discuss?

In my opinion the compiler should be able to produce also separate bundles, like this:

{
    "...": "...",
    "moduleDeclarationOutput": "./modules",
    "modules": {
        "module-a": {
            "from": "./src/module-a/index.ts"
        },
        "module-b": {
            "from": "./src/module-b/index.ts"
        }
    }
}

having then in ./modules/module-a.d.ts the definitions of types exported by ./src/module-a/index.ts and respectively for "module-b".

[davismj]

requesting an update on this please

[mhegazy]

nothing done since last update. no plans to do anything for TS 2.0 at this point.

[davismj]

@weswigham What do you think would be a best strategy in light of this? Would you recommend building a custom TypeScript with your PR?

[weswigham]

I wouldn't recommend it, no. TBH, with recent changes to allow module
augmentation and even proposals for relative paths in module augmentation,
we may have a better way to do this now which can even preserve secondary
entry points.

On Mon, May 16, 2016, 6:24 PM Matthew James Davis notifications@github.com
wrote:

@weswigham https://github.com/weswigham What do you think would be a
best strategy in light of this? Would you recommend building a custom
TypeScript with your PR?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#4433 (comment)

[davismj]

Any other options, then?

[matthew-dean]

Note that rollup-plugin-dts doesn't work with TS path aliases, making it a non-solution. rollup-plugin-ts works, but isn't compatible with TypeScript 4.7-4.9. So basically there are no solutions right now, if someone uses paths aliases. It's not great that if you run TSC to generate types, it will create .d.ts files that do not resolve paths, making the output unusable for distribution. TSC + paths is essentially broken by default.

[matthew-dean]

@weswigham

We consider API Extractor and similar tools to be good community solutions to this and don't really see a pressing need for a builtin one, so I'm going to close my own proposal now. sparkling_heart

There definitely are no good community solutions to this. Have you seen the size of the configuration file for API Extractor, just to bundle .d.ts? Even with that overhead, in many cases it just doesn't work.

[wdanilo]

We consider API Extractor and similar tools to be good community solutions to this and don't really see a pressing need for a builtin one, so I'm going to close my own proposal now. 💖

What do you mean by "good community solution"? After fighting for an hour with it, I gave up. These are just two of MANY errors I got:

[Hotell]

What do you mean by "good community solution"? After fighting for an hour with it, I gave up. These are just two of MANY errors I got:

That's a known issue which has a workaround microsoft/rushstack#2780. But it's indeed concerning that it doesn't follow whole ECMA syntax

[octogonz]

@wdanilo Just curious, what is on line 58 of task.ts?

Also, what do you get with the --diagnostics CLI parameter?

[matthew-dean]

@wdanilo Yeah API Extractor is pretty bad

[jsejcksn]

Ref: TypeScript’s Migration to Modules > Preserving Our API and Bundling Our Declaration Files

The referenced bundler module (latest commit at the time the article was published): https://github.com/microsoft/TypeScript/blob/3f4d16a25ed82c43d06606da3e2001efc82b58eb/scripts/dtsBundler.mjs

[jakebailey]

I cannot express enough how much nobody should use, let alone look at that bundler. Unless your project is TypeScript itself, it almost assuredly will not do the right thing, nor do I even think what it's doing is a good example of how a usable dts bundler should be written.

[joelday]

@jakebailey tell us how you really feel.

[jsejcksn]

@jakebailey 😄 I suppose most won't read that article anyway, so here's a snippet — it seems empathetic to several of the sentiments expressed previously in this issue:

Just like there are many options for bundling JavaScript, there are many options for bundling .d.ts files: api-extractor, rollup-plugin-dts, tsup, dts-bundle-generator, and so on.

These all satisfy the end requirement of "make a single file", however, the additional requirement to produce a final output which declared our API in namespaces similar to our old output meant that we couldn’t use any of them without a lot of modification.

In the end, we opted to roll our own mini-d.ts bundler suited specifically for our needs. This script clocks in at about 400 lines of code, naively walking each entry point’s exports recursively and emitting declarations as-is.

[jakebailey]

If TypeScript itself had a d.ts bundler, I can assure you that the need for dtsBundler.mjs would not have gone away. The TypeScript compiler predates modules and thus its API was declared using namespaces. When we switched to modules, we had to make a conscious effort to construct our code in a very goofy and restrictive way just to make sure that it matched that format, then write custom code to massage the d.ts output into something that looked the same to our external users. If TS had been modules from the start, none of this would have happened; I doubt I would have even written dtsBundler.mjs. We'd probably be shipping unbundled.

Personally, I think a much more reasonable future is going to be one with #47947/#53463 (or similar, in TS or not), which would more explicitly allow bundlers like esbuild to do what they do for JS source but on types; all of the same considerations about hoisting, renaming, and API shape reconstruction are already things that bundlers have to contend with.

[lgarron]

I thought I'd mention another frustration that can't be resolved by using third-party bundlers: it is difficult or impossible to test new TypeScript versions until significantly after they are released.

TypeScript 5.1 implements an important feature for one of our projects. But because we're using a type bundler that only supports up to TypeScript 5.0, I cannot install typescript@v5.1 without causing npm to freak out (and refuse). I could override that, but npm install would still fail for the repo by default.

Despite being careful to maintain a 100% vanilla TypeScript project without bells or whistles, this leaves us stuck waiting a few weeks or months to hope that other projects update their support¹, before we can fully test that our project works properly with the TypeScript 5.1 feature we need. This is a rather frustrating experience, and would be avoidable if TypeScript had some sort of reference implementation for type bundling that we could use.

Footnotes

1. I could probably contribute to those projects, but I have about half a dozen much more urgent contributions for other projects that I already need to prioritize. ↩

[timocov]

@lgarron I understand your frustration, but it could be applied to any library you use. Or for example linter - it is possible that it might be slightly behind current compiler version, but it doesn't mean that it should be part of the compiler. I feel like this problem needs to be solved by package's maintainers/community support (e.g. the type bundler I maintain has tests running against typescript@next in CI to make sure that everything is alright in the next version and they also are running weekly in case of not having commits to the tool but new compiler releases) rather than merging tools together.

[unional]

IMO it's essentially the difference between two approaches: the NodeJS/JavaScript distributive approach, vs the GoLang/Rust+Cargo/Bun approach.

[lgarron]

@lgarron I understand your frustration, but it could be applied to any library you use. Or for example linter - it is possible that it might be slightly behind current compiler version, but it doesn't mean that it should be part of the compiler.

I understand that this is the case for project maintenance tools in general, but I think this misses my point. When I publish a TypeScript-based library that is easy for everyone in the ecosystem to use, I'm responsible for producing two transformations of the source code:

• The JS
• The types

Those are what it's all about — they constitute the library. Things like linting help with project maintenance, but do not generally affect these outputs for any given input code.

I'm advocating that the reference tsc compiler should be able to produce a publishable library for vanilla TypeScript project, and that type bundling is an important feature of this.

After all, the main point of TypeScript is to provide developer ergonomics through the type system, and unbundled type output is not very ergonomic. For the main project that I maintain (https://github.com/cubing/cubing.js), npx tsc outputs 359 .d.ts files. To understand the source of some types, you'd have to look through half a dozen files that are filled with many types that should not be visible outside the project¹. By contrast, tsup outputs 17 well-crafted files that include just what someone using my library needs.

I fact, I think there's a good argument that that ergonomically bundled types are more important than bundled .js for people using a published library: When you look up a symbol from a library that you're using, you are generally looking at its type files — "Go to Definition" has done this in VSCode for as long as I know, whereas "Go to Source Definition" was only recently added. The .d.ts files are essentially a developer interface, in a way that the .js files often are not.

My comment above was meant to point out that it is not quite practical to build and test publishable library files (JS files and type files) until significantly after a TypeScript release. This makes it harder to prepare our code, and to contribute feedback to the TypeScript project about using upcoming language changes in real-world code.

I feel like this problem needs to be solved by package's maintainers/community support

I think this is a reasonable stance in principle. However, the large amount of upvotes and comments in this thread make it clear that the ecosystem has only developed limited solutions after a decade of TypeScript — hence our advocacy for the TypeScript compiler to take the lead and adopt it as a feature.

Footnotes

1. Some of which may have names that are deceptively similar to ones that should be visible. ↩

[llllvvuu]

When you look up a symbol from a library that you're using, you are generally looking at its type files — "Go to Definition" has done this in VSCode for as long as I know, whereas "Go to Source Definition" was only recently added. The .d.ts files are essentially a developer interface, in a way that the .js files often are not.

Unless there is .d.ts.map, which brings up another point: dts-bundle-generator, api-extractor, and rollup-plugin-dts all do not support .d.ts.map. Additionally, api-extractor and dts-bundle-generator do not have chunking, while rollup-plugin-dts is in maintenance mode and author is in this thread suggesting for this to be in tsc.

dts-buddy is an exciting WIP here. I'm not sure if I like the declare module vs having entrypoints be physical files as per "exports" map (not supported in older Node I'm aware).

Currently I think the best DX for a package might be provided by using tsc with "declarationMap": true and no bundling of the .d.ts, but yes bundling of .js. The good: types and go-to-source work, and the non-exposed exports will still be hidden at runtime (and even at compile-time if "moduleResolution": "nodenext"). The bad: declarations for un-exposed exports will still be in the files, so the download size will still not be ideal.

If we build our code with something like tsc --build tsconfig.packages.json, there's no room to use any bundler: when tsc tries to build b, it's expecting a types to be there.

This seems like maybe the strongest counterargument to having .d.ts bundling be in userland. One could try having the bundler work in-place, but that feels iffy.

[imjuni]

I hope this issue will be discussed again. When we do library projects in TypeScript, we need to generate and bundle .d.ts files for reuse. Since tsc doesn't support this process, we have to find another tool, and there are a lot of them, but they all behave a little differently, so it's a pain to sort through them for our purposes.

Since tsc does not apply the path re-map and absolute path(eg. 'src/modules/Button') to the output when generating a .d.ts file, bundling is impossible without the help of other bundling tools. This is making my experience of using TypeScript in library projects a bad one. For example, Svelte has decided to go back to using JavaScript, and my guess is that it's because of the above.

I think what is needed is for tsc to provide this functionality natively, or for dts-gen to generate bundled .d.ts files using tsconfig.json like tsc does.

[ivancuric]

This is still an extremely frustrating situation.

I have a monorepo project for a library that consists of the main entry point package and a web worker. The web worker is its own npm package as it's a different compile target (iife), won't be exposed (internal package), and has a different lib mode ("lib": ["esnext", "webworker"]) compared to the main package.

Most of the types are in the worker package, and it's a peer dependency of the main package, in order to inherit the types.

The problem is that when I bundle the package, I want to expose the types from the web worker as well. However, with tsc I only get stuff like export type {a} from 'unpublished-worker-package.

So then I need to use api-extractor, output the unnecessary declaration files for the worker and hope it will do its job.

[JPPereira123]

Agreed, we've got an npm workspace that is an API client that imports types from another workspace (that is just the API layer that will never be published), so we need to bundle its types as part of the client tsc build.