Generate a single declaration file for external modules

[mhegazy]

If compiling external modules, with --declaration, each module gets its own declaration file. this is not useful for library authors, who probably want to hand off a single .d.ts file for all their modules. the only way to do this now, is to either hand edit the file, or use a tool to concatenate the output.

This becomes even more desirable with ES6 module export/import where a module can reexport exports of another module.

See discussion in #2516 (comment) for more details.

[mhegazy]

Also related to #17

[basarat]

Tagging @csnover because of https://github.com/SitePen/dts-generator

[NoelAbrahams]

I would like to have a generate-single-declaration-file option (without combining JavaScript output) for _internal modules_ as well. See #192.

[ToddThomson]

My experiences with Typescript 1.5 have been positive. The only thing that keeps giving me grief is the internal "module" ( which is only a namespace ). I really don't like fighting back and forth between internal and external/real/es6 modules and what I am allowed to do with both ( mostly the grief that comes from defining external module bundles ). I really only want to use the ES6 module syntax going forward.
I have no issues with 1 file = 1 es6 module and optimizing during a production build. However there are times that I want to bundle files/modules together at compilation time. Even if this just produced a named typescript definition file I would be happy.
I am wondering if you can leverage the tsconfig.json "project" file to provide "bundles". A bundle would look like:
"bundles": {
"bundle-name": [ "some/module", "another/module" ],
"another-bundle-name": [ "yet-another-module" ]
}
An appropriate arg could be used for command line builds.

[3nsoft]

@mhegazy I want to re-phrase your statement.
Every lib developer wants an auto-magic generation of declarations. Period.

If all code initially uses internal modules, then it does not matter if declarations are spread over several files. Files can be copy-pasted and used as is. I see no problem in this scenario.

But, if your lib uses external libraries, i.e. import/require, then manual labour is required, because developer needs

declare module "external-thingy" {
...
}

and this currently is not done automatically.
Manual process involves writing "internal module" declaration, which is "export = Foo" into external module declaration. We need are way to have simple (automatable) solution to declare above thing.
Again, number of files is not an issue!

[3nsoft]

If modules are all internal, then multiple files declaration can be concat-ed with gulp / grunt into single file without stepping on each other's toes. Such is simple nature of internal modules.

Say. There was a news that number of node (npm) modules is already comparable if not surpassing a number of maven (!) modules. Hooray! It is time to fix external modules' declarations!

[ToddThomson]

I've put together an npm module ( TsProject ) that uses the tsconfig.json file to define bundles. It generates a typescript source file based on the dependencies in a source file as specified by a bundle. The generated typescript source file is output along with the compiled javascript and definition file.

[poelstra]

I'm not too sure it's really "a single .d.ts file" that we'd need.

Having one .d.ts file per .emitted .js file is a very good fit for the CommonJS way of working.
Relevant advantages I see:

• possibility for having multiple versions of the same module
• only rewriting changed files, without having to rewrite the complete bundle

Note that with external modules, the name of an external module is typically not present in the file itself. It is determined by its filename. So, I'd vote for keeping the currently generated .d.ts files as-is, and not e.g. put 'declare module "my-module" {}' inside of them, not even in an 'overall bundle file'.

Tools like dts-bundle and dts-generator seem to prefer to write out one file, but I believe this is mainly because TypeScript doesn't allow specifying relative paths for the import()'s in .d.ts files.

I think that if TSC would allow those paths even in .d.ts files, dts-bundle or dts-generator would no longer be necessary.

Still, a solution needs to be found for any other 'really external' modules (i.e. different npm packages, not just different .ts files inside the package) that are referenced, as this would basically mean that the /// <reference ...> line of the package would need to be 'local' to that package's .d.ts file. Otherwise these really external modules will start conflicting with similar modules in other packages.

TSC disallows having an external module be declared multiple times, which you can only get to work if all references are pointing to the same definition file. So, let's not try to 'throw away' that information by bundling, but support the case of leaving the files where they are, and use the 'real' filesystem to resolve the references.

Note: my use-case is mainly 'plain' CommonJS, using TSC's --outDir option, and using browserify for the browser. I guess that the single-.d.ts-file case would be handy for TSC's --out option.

[3nsoft]

Use case against clobbering declarations into a single file.

Step one. I ship a node lib with one declarations file. Due to having declarations, users of the lib (developers!) now can see type info and comments, when these were properly transferred into declarations.
Step two. In Eclipse, for example, developer is pressing F3 to see what is used. IDE will show only respective definition.
Developer's intention, though, is to see a source (js) of a beast, which he is using. If declarations' folder structure maps that of module's code, then respective js can be easily found. Not so, when declarations are in a single file.

[basarat]

Worth sharing here : #2829 all it needs is the path to your package.json + tsconfig.json to generate and consume node packages. no additional configuration required.

[basarat]

@poelstra

my use-case is mainly 'plain' CommonJS, using TSC's --outDir option, and using browserify for the browser. I guess that the single-.d.ts-file case would be handy for TSC's --out option

Not true. Look at : https://github.com/TypeStrong/atom-typescript-examples/blob/master/node/lib/main.ts You can import any file you have exported despite using a single .d.ts definition file

Again the details of the proposal : #2829

[poelstra]

@basarat I know about that possibility, yes, and in fact, that's what we use today (using output of dts-bundle at work, and recently started using dts-generator at home).

The problem I see with it though, is that it will break as soon as more than one dependency level is introduced.

Consider this dependency tree:

• myprogram
  • mylib
    • myutils@1.0
  • myotherlib
    • myutils@2.0

In the case of generating .d.ts files with declare module "myutils" { ... }, etc, all of these declarations will be in the same 'global module name namespace'. Thus, mylib and myotherlib cannot each use their own version of myutils.

This is not a problem as long as myutils is only used internally (and thus its types are never exposed through mylib/myotherlib), but that will often not be the case.

[basarat]

@poelstra awesome. Perhaps you can hash out what a .d.ts would look like for mylib/myotherlib/myutils@1.0/ myutils@2.0

Then I can have something to work from and understand your proposal better (perhaps then I can even implement it).

[poelstra]

[edit]

TL;DR re-ordered summary of the below:

• Keep existing .d.ts output, i.e. one declaration file per .ts file, and not wrapped in declare module "..."
• When looking for the .d.ts corresponding to an import, use the same logic as Node currently does when it looks for .js files
  • e.g. looking at package.json's main field for a simple require("mypackage"), but also allow the require("mypackage/relative/path") style

[/edit]

@basarat do you ever sleep? You seem to be online 24/7 😄

Strange as it may sound, my proposal would be to largely keep the .d.ts files as they currently are generated by tsc: i.e. without declare module "..." {}, and with explicit import("./somefile") inside of them.

Changes to the compiler would be that it:

1. uses 'the same' resolution logic that node.js uses for finding the .js files, but then for finding the .d.ts files (I think I saw an issue for that somewhere already), and allows looking for accompanying .d.ts files both 'next to' that .js file, and in e.g. the typings/ folder of the importing package (the importer, not importee).
2. allows (and resolves) import("./relative/path") statements inside the .d.ts
   3a. makes /// <reference> lines 'local' to the .d.ts files (and the .ts files themselves)
   3b. we stop using declare module "..." altogether (i.e. the external-module version, not necessarily the internal non-quoted version)

Point 3a. is probably the most problematic one, as people have come to expect these to be project-wide, and it's even actively promoted for some cases, IIRC to have a more C#-like experience.

Point 3b. might be a nice alternative: the compiler would then search for (DefinitelyTyped) definitions in e.g. a typings/ folder, and in that folder would be typings exactly as they're generated for 'normal' external modules. The trouble is that it would basically require every existing external-module typing on DT to be updated (i.e. the declare stripped from it)...

Example:

/// src/bar.ts
export interface BarType {
  bar: number;
}

/// src/foo.ts
import bar = require("./bar");
export function something(): bar.BarType {
    return { bar: 42 };
}

/// src/tsconfig.json
{
    "version": "1.4.1",
    "compilerOptions": {
        "target": "es5",
        "module": "commonjs",
        "declaration": true,
        "noImplicitAny": true,
        "removeComments": false,
        "noLib": false,
        "outDir": "../dist"
    },
    "filesGlob": [
        "./**/*.ts",
        "!./node_modules/**/*.ts"
    ],
    "files": [
        "./bar.ts",
        "./foo.ts"
    ]
}

Existing typings generated by Typescript:

/// dist/bar.d.ts
export interface BarType {
    bar: number;
}

/// dist/foo.d.ts
import bar = require("./bar");
export declare function something(): bar.BarType;

(In atom-typescript, opening dist/foo.d.ts gives me Cannot find external module './bar' on line 1, which I guess is because TypeScript doesn't 'understand' its own .d.ts?)

I like these generated typings very much, as they are a direct reflection of the sources. Now if only we could actually use them :)

Let's call this package myutils. Now, suppose that I have main: "dist/foo.js" in package.json, I'd like the typescript compiler to automatically use dist/foo.d.ts when I import bla = require("myutils") in e.g. mylib.

Another example would be if myutils is not a TypeScript package.
I'm not completely sure how it would look on DefinitelyTyped, but I imagine it could look like:

DefinitelyTyped/myutils/latest/dist/bar.d.ts
DefinitelyTyped/myutils/latest/dist/foo.d.ts
DefinitelyTyped/myutils/1.0/dist/bar.d.ts
DefinitelyTyped/myutils/1.0/dist/foo.d.ts

The contents of these files are exactly like given above (i.e. no declare ...).
In your local package directory (e.g. mylib/typings), you'd have just myutils/dist/bar.d.ts and myutils/dist/foo.d.ts. Now, TypeScript would look at the package.json of myutils, which points to dist/foo.js, and it knows to search the typings folder for myutils/dist/foo.d.ts.

Note that in practice, most packages include some kind of index.js (and thus index.d.ts).

Also note that it still isn't required to exactly replicate a package's file-structure. As long as a package e.g. exports all/most of its (public) types through its 'index' (which most packages do), having everything in one .d.ts file works perfectly fine. So it really should be 'just' stripping the declare statement in most cases.

In both scenarios (TS and non-TS package), there is no declare module "myutils" anywhere, so different versions of myutils can be included by mylib and myotherlib. (Just like I could have two foo.ts files, as long as they are in different directories.)

[3nsoft]

@poelstra your proposal may emotionally seem strange, because it is eery close to the simplest, yet good, solution. I love what you propose, and only want to simplify it a little further.

1. Leave generation of declarations from node modules as it is, i.e. without adding declare module "foo" { ... }, and without bundling everything into one file.

2. When ts compiler cannot find declarations for an npm package in already existing places (whatever they are now), look into npm's package.json, where it should understand a field like typescript.declarations.main: string, which should point to declarations as they are currently produced, and ts compiler should understand them.
   Notice that declarations.main corresponds to package's main script.
   Highlight that this lookup should happen after existing ways for finding declarations. Firstly, this will allow for a graceful change from relying on DefinitelyTyped to using this new way. Secondly, it allows a nice way to overshadow declarations, when one wants to screw with them.

This is it. Just two points, and nothing else needed!

We should not change anything with /// <reference>. At least I do not see any reasons for it. And here is why.
In the future world, when above proposal is implemented, one will not need to use /// <reference>. Really. Yet, its current over-reaching power may come handy at some holy-s&&t build moment. For example, when gulp-ing, tsify plugin takes in one file. So, a rescue for my browser build is /// <reference>. Alternative, with gulp-typescript, for example, does not require such hack. More so, consumers of my package may have this comment in js, but it is harmless, as it is a mere comment in js. Note that for node build, or inside modules I do not use /// <reference> at all. And IDE, Eclipse with ts plugin, sees all definitions in the project anyway, so that /// <reference> is again not needed. Yes, it is amazing how much docs talk about it, and how little I actually needed it. But when this tool is needed, its better have its current power.

@poelstra TS 1.5 does generate for me .d.ts with relative imports. I humbly assume that, if compiler makes it, compiler must consume it. If this is not so, we have a bug for @mhegazy

@mhegazy can you comment yourself on this proposal and pass it to other gurus and makers of a wonderful TS.

• Proposal contains simple solution to an existing pain.
• Solution is additive in nature and seems to fit, at least when viewing TS as a magical black box.
• There is this pleasant outcome that declarations also follow ES6 mantra of one-file == one-module.

[3nsoft]

Personal preferences, five cents.

I want to get rid of tds.d.ts. I want to simplify things.
Please, do not make me use yet another config file like tsconfig, or whatever you make me remember. Please. Do not make me-e-e!!
Put it into existing package.json. Imagine, developer looks at it and immediately knows that TS can be used with your package. Or, he will google a strange new field in package's file: typescript.

[poelstra]

@3nsoft Nice writeup, mine indeed sounds way more complicated than what is actually proposed :)

I've editted my earlier post with a TL;DR for future readers...

Note that I like your idea of using package.json's main field, but I think it doesn't support the require("mypackage/some/path") case as neatly as letting the compiler replicate Node's search.

Re: changing /// <reference>: yeah, I don't want to change it (not for this, at least ;)). One still 'needs' it to e.g. make mocha stuff available though.

[mhegazy]

@3nsoft,

1. Leave generation of declarations from node modules as it is, i.e. without adding declare module "foo" { ... }, and without bundling everything into one file.

This is the current behavior now, each .ts file will generate a single .js file and a single .d.ts files. the compiler never emits module declarations in this form. these are only hand-authored.

2. When ts compiler cannot find declarations for an npm package in already existing places (whatever they are now), look into npm's package.json, where it should understand a field like typescript.declarations.main: string, which should point to declarations as they are currently produced, and ts compiler should understand them.

This is tracked by #2338, and @vladima is working on now. so should be available for you to try soon.
The change should allow the typescript compiler to follow a resolution logic that matches node's resolution logic, and adds a "typings" field to match "main" in the package.json. please take a look at the proposal, and do let us know if there is any thing you would want to see different. your feed back is always appreciated!

@poelstra and @3nsoft , i think your proposals are valid, and needed and long over due. I believe #2338 captures this proposal, in addition to AMD module loading cleanup as well.
Bundling is however a slightly different story. You should look at #2743 for the .js bundling side of the issue, this issue should handle the .d.ts underselling.

As a library author i would like to have control over the shape of my externally visible .d.ts file. We have the same issue for the TypeScript package. we do some tricks to hide some declarations that we do not want to be public but we would like to share between different modules. Bundling gets specifically interesting with ES6 modules, where you can do a lot of compositions with export * from "mod" syntax. With this new syntax you end up with small modules that reference each other..
consider a project for this form:

// main.ts
export * from "core\core";
export * from "utilities";

// core\core.ts
export * from ".\part1";
export * from ".\part2";
export * from ".\part3";
export { helper } from ".\part4\helper;

// utilities.ts
export { getName } from "tools\helpers";


// core\part2.ts
export function build() {
}

you really do not want your users to look at this .d.ts and keep jumping between exports to know that your module has a "build" function. I think for these scenarios you want the compiler to digest your project, and spit out a single module .d.ts that tells users what is the shape of the module and hides all the unnecessary implementation details:

// main.d.ts
declare function build(): void;
declare function getName(): void;
declare module helper {}

[3nsoft]

@poelstra when I say that typescript.declarations.main corresponds to usual main, I mean that for package's main js script compiler should use a declaration identified by typescript.declarations.main. And this field may point to whatever. It can be, in my case, compiler generated tree of declarations. Or in JQuery's case, it can be a single file, reused from DefinitelyTyped repo.

[3nsoft]

@mhegazy yap. #2338 is exactly this 👍 . Indeed, it is funny how in "available soon", the word "yesterday" is spelled with just four letters: "soon". Weird. :)

[poelstra]

@mhegazy Aaaaah, I see. Makes sense, thanks! :)
#2338 is indeed what I was really after.

Agree that ES6 modules indeed makes this 'bundling' more useful for human consumption as a kind of documentation.

[oliverw]

Reading this thread and the discussions related to issues referenced by this one I'm still not any wiser what the final solution will be. Any suggestions? :)

[mhegazy]

The issue has gone a bit long. I will log a new issue with some more details once i have them :) the basic idea is a command line switch, that tells the compiler to bundle the .d.ts from different modules into a single .d.ts file

[tommyZZM]

👍

[davismj]

Is this issue pending? Is it going to be dropped? Has it been implemented but not documented?

[mhegazy]

This should be marked as completed. with #5090, declaration output can be bundled.
for flattening the declaration, i.e. removing declarations that are not exported from an entry module, #4433 is the correct issue moving forward.

[bennycode]

@mhegazy, how can declaration output files be bundled?

I followed up on #5090 and there it's mentioned that you need to set --module along with --outFile. Unfortunately this doesn't work with "module": "commonjs" because of:

error TS6082: Only 'amd' and 'system' modules are supported alongside --outFile.

[mhegazy]

why do you want to bundle your declarations if you are using node? the .js files are not bundled either.

[Delagen]

@mhegazy if TS itself cannot bundle JS files, that does not mean that no any other tool.
JS files can be bundled with rollup, webpack or any other bundler.
For now I bundle declarations using typings tool, that works, but result all declarations in file, not only exported by main entry. Result declaration bundle contains from many declare module sections with add main with package.json module name. But this is more convenient for me instead of place one compiled module (sometimes with sourcemap file) in package uploaded to registry with many d.ts files generated by TS compile.

[bennycode]

@mhegazy I am getting lost. Maybe you can help me.

I have a TypeScript application with the following configuration (tsconfig.json):

{
  "compilerOptions": {
    "declaration": true,
    "lib": ["DOM", "ES6"],
    "module": "CommonJS",
    "moduleResolution": "node",
    "noEmitOnError": true,
    "noImplicitAny": true,
    "noImplicitReturns": true,
    "outDir": "dist/commonjs",
    "removeComments": true,
    "rootDir": "src/ts",
    "sourceMap": false,
    "target": "ES5"
  },
  "exclude": [
    "dist/commonjs",
    "node_modules"
  ]
}

When I run tsc, then the TypeScript compiler creates several JavaScript files (*.js) and several TypeScript definition files (.d.ts):

Now I want to distribute my code as npm package and I would like to support types for people using my code but I don't know where I should point the types property in my package.json to.

The documentation says: "Set the types property to point to your bundled declaration file" but I don't have a definition bundle (like main.d.ts or index.d.ts). Instead I have several *.d.ts files. 😕

So what can I do now to publish my npm package along with it's definitions?

[Delagen]

@bennyn You can publish it as is. (all .js and .d.ts files)

[orefalo]

@bennyn I have the same issue
@Delagen How? file: [ "dist/" ] ?

[ikatyang]

@bennyn @orefalo

For short, types is the main for types.

---

Just like the main field in package.json, you have to specify the declaration file for the main file, then TS will follow the module resolution (import/export) to find those related type.

For example:

(package.json)

{
  "main": "lib/index.js",
  "types": "lib/index.d.ts"
}

[Delagen]

@orefalo if file d.ts place is near the main js the types section is unneded. TS compiler try to load declaration with the JS file loaded, so no action needed.
If declaration file is different named than main file, so it must be specified in types property of package.json.

[bennycode]

@ikatyang I have done what you have suggested. In my library I pointed types to my index.d.ts file:

package.json

"types": "./dist/commonjs/index.d.ts"

./dist/commonjs/index.d.ts

import WireAPIClient from './core/WireAPIClient';
export = WireAPIClient;

Afterwards I tried to use my library in another TypeScript project:

import {WireAPIClient} from '@wireapp/api-client';

This caused the following error:

error TS2497: Module '"D:/dev/projects/bennyn/wire-web-core/node_modules/@wireapp/api-client/dist/commonjs/index"' resolves to a non-module entity and cannot be imported using this construct.

So I changed my import statement:

import WireAPIClient = require('@wireapp/api-client');

Which then led to the following errors:

node_modules/@wireapp/api-client/dist/commonjs/auth/AuthAPI.d.ts(15,30): error TS2304: Cannot find name 'LoginData'.
node_modules/@wireapp/api-client/dist/commonjs/auth/AuthAPI.d.ts(16,22): error TS2304: Cannot find name 'LoginData'.
node_modules/@wireapp/api-client/dist/commonjs/auth/AuthAPI.d.ts(16,42): error TS2304: Cannot find name 'AccessTokenData'.
node_modules/@wireapp/api-client/dist/commonjs/auth/AuthAPI.d.ts(18,27): error TS2304: Cannot find name 'AccessTokenData'.

LoginData and AccessTokenData are classes from my library and they are in the same directory as AuthAPI, that's why there is no import statement in AuthAPI for them. So it seems like the TS module resolution doesn't work properly in my case?

[ikatyang]

@bennyn I have no idea what's going on to your compiled dts file, since I cloned your PR and execute yarn run dist, I found that your compiled AuthAPI.d.ts somehow miss the triple-slash directives.

(AuthAPI.d.ts)

+/// <reference path="LoginData.d.ts" />
+/// <reference path="AccessTokenData.d.ts" />

It compiled fine for me (with triple-slash directives exist), and thus the following import works fine.

import WireAPIClient = require('@wireapp/api-client');

NOTE: There are several unlinked definitions in your code (does not use import or triple-slash-directives, so TS cannot find them), you have to correct them to get the result.

[bennycode]

@ikatyang Thank you so much for your support!

I forgot to regenerate the .d.ts files in the @wireapp/api-client project. After I did that everything was running smooth. I've also found a better alternative to the triple-slash directives:

Before (no interface export + triple-slash)

LoginData.ts

interface LoginData {
  email: string;
  password: number | string;
}

AuthAPI.ts

/// <reference path="LoginData.d.ts" />
import {AxiosPromise, AxiosRequestConfig, AxiosResponse} from 'axios';

export default class AuthAPI {
  constructor(private client: HttpClient) {}

  static get URL() {
    return {
      COOKIES: '/cookies',
      INVITATIONS: '/invitations'
    };
  }

  public postCookiesRemove(login: LoginData, labels?: string[]): AxiosPromise {
    const config: AxiosRequestConfig = {
      data: {
        labels: labels,
        password: login.password.toString(),
      },
      method: 'post',
      url: `${AuthAPI.URL.COOKIES}/remove`,
    };

    return this.client.sendRequest(config);
  }

  // More code follows here...
}

After (interface export + interface import)

LoginData.ts

interface LoginData {
  email: string;
  password: number | string;
}

export default LoginData;

AuthAPI.ts

import LoginData from './LoginData';
import {AxiosPromise, AxiosRequestConfig, AxiosResponse} from 'axios';

export default class AuthAPI {
  constructor(private client: HttpClient) {}

  static get URL() {
    return {
      COOKIES: '/cookies',
      INVITATIONS: '/invitations'
    };
  }

  public postCookiesRemove(login: LoginData, labels?: string[]): AxiosPromise {
    const config: AxiosRequestConfig = {
      data: {
        labels: labels,
        password: login.password.toString(),
      },
      method: 'post',
      url: `${AuthAPI.URL.COOKIES}/remove`,
    };

    return this.client.sendRequest(config);
  }

  // More code follows here...
}

Outcome

With export default LoginData; I can now use the @wireapp/api-client library like this:

import Context from "@wireapp/api-client/dist/commonjs/core/Context";
import LoginData from "@wireapp/api-client/dist/commonjs/auth/LoginData";
import WireAPIClient = require('@wireapp/api-client');

const client: WireAPIClient = new WireAPIClient({
  rest: 'https://prod-nginz-https.wire.com',
  ws: 'wss://prod-nginz-ssl.wire.com'
});

const credentials: LoginData = {
  email: 'name@mail.com',
  password: 'secret',
  persist: false
};

client.login(credentials).then((context: Context) => {
  console.log(`Got self user with ID "${context.userID}".`);
});

This is exactly what I wanted to do. Thank you very much! 🍻