External module resolution for non-Typescript packages

[poelstra]

Good progress is made on supporting the case of packages that carry their own typings in #2338.

That proposal doesn't currently deal with modules that do not carry their own typings, a.k.a. all modules that need typings from DefinitelyTyped.

This issue is to investigate possibilities of, and propose a (simple) solution for preventing problems when using (multiple versions of) external modules together.

TL;DR

1. No longer use declare module "..." { } in typings for external modules
2. Add an extra step to the resolution logic of External module resolution logic #2338 to look up this type of external typings

Example scenario

• myprogram
  • mylib
    • myutils@1.0
  • myotherlib
    • myutils@2.0

Suppose that both mylib and myotherlib export a function, that returns a value of a type defined in myutils.

Existing scenario

Likely, both mylib and myotherlib will each have a /// <reference path="./typings/tsd.d.ts" /> line, which again has a /// <reference path="./myutils/myutils.d.ts" /> line.

Problem

Both libs carry their own version of myutils.d.ts, which works fine as long as they are not used together.

However, when they are used together (as in this example), all the /// <reference ...> lines will (need to) be 'merged', which essentially means the declare module "myutils" { ... } from both mylib and myotherlib will end up in the same 'global module name namespace'.

This gives a compiler error, and even if the compiler would somehow allow it, would make it impossible to refer to either version of myutils.

Additionally, tools like dts-generator e.g. include the /// <reference> line inside of the declare blocks, which get ignored by the compiler. This leads to unknown-references when trying to use such a typing, and will typically be solved by adding e.g. a myutils/myutils.d.ts to myprogram/typings/tsd.d.ts, which will be wrong for one of the libs.

Solution

External modules have the nice property that there is no 'global namespace' (except the filesystem, maybe), and basically every reference is 'local'.

It makes sense to use the same referencing scheme that's used for resolving 'local' external modules (within a package) for 'normal' external modules (different npm packages).

Basically, the idea consists of:

• No longer using declare module "..." { } in typings for external modules (to prevent the global namespace clash)
• Adding an extra step to the resolution logic of External module resolution logic #2338 to look up this type of external typings (e.g. in typings/ folder)

Advantages

• Multiple versions of a package can co-exist in the same program/library
• /// <reference> lines are no longer needed in most cases (except probably for e.g. importing Mocha's it() and other 'true' globals)
• One format to rule them all: external module typings are already generated without declare module "..." { } by tsc, so all 'sorts' of external module typings will look the same
• Gradual upgrade path: the tsd.d.ts-way will still work for packages that do still use declare module "..." { } (although it will still have the name-clash problem etc.)

Disadvantages

• Typings on e.g. DefinitelyTyped will (gradually) need to be replaced with their non-wrapped version
• Extra lookup step in the compiler
• Location of these external typings (a.k.a. include path) needs to be configured somewhere and/or sensible default value chosen

Example 'new-style' typings

Most hand-crafted typings will typically look like:

/// myutils.d.ts
export interface BarType {
    bar: number;
}
export declare function something(): BarType;

But just to make things more interesting, suppose the typings of myutils look like:

/// dist/bar.d.ts
export interface BarType {
    bar: number;
}

/// dist/foo.d.ts
import bar = require("./bar");
export declare function something(): bar.BarType;

Nothing special here, this is what tsc generates today.
(Note: dist/ may be because someone used CoffeeScript, not TypeScript, right? :))

Now, the typings dir on e.g. DefinitelyTyped could look like:

DefinitelyTyped/myutils/latest/dist/bar.d.ts
DefinitelyTyped/myutils/latest/dist/foo.d.ts
DefinitelyTyped/myutils/1.0/dist/bar.d.ts
DefinitelyTyped/myutils/1.0/dist/foo.d.ts

(Or maybe 2.0 instead of latest, ideas welcome.)

And both mylib and myotherlib would have:

typings/myutils/dist/bar.d.ts
typings/myutils/dist/foo.d.ts

Looking up .d.ts

Given e.g. import { something } from "myutils" the compiler could use myutils/package.json's main property to get to ./dist/foo.js, which in turn would resolve to dist/foo.d.ts.

Nice and simple: no need to have support from the authors of myutils, no /// <reference> lines.
And it even supports the import bar = require("myutils/dist/bar") case.

Open questions

• Use one typings dir for both 'wrapped' and 'unwrapped' typings?
• First do a lookup for external modules already declare'ed, e.g. through tsd.d.ts? If so, would probably make first bullet usable, nice for gradual upgrade path of DT typings.
• Most DT typings will not want to mimic exact filesystem structure of package, using e.g. myutils/latest/index.d.ts will probably work out-of-the-box with external module resolution logic, but myutils/latest/myutils.d.ts or even myutils/myutils.d.ts may be easier for filename in editor tab?

Good idea? Bad idea?

[basarat]

My hacky suggestion:

• myprogram
  • mylib
    • myutils@1.0
  • myutils@2.0

In this case again if we only read myutils@2.d.ts and mylib.d.ts then there would be no issue. But since mylib.d.ts references myutils@1.d.ts we will have a conflict. My proposal is a conflict resolution algorithm such that:

• myprogram
  • mylib
    • myutils@1.0
  • myutils@2.0

We only read the following files into the compilation context of myprogram : [mylib.d.ts,myutils@2.d.ts]

---

Note in:

• myprogram
  • mylib
    • myutils@1.0
  • myotherlib
    • myutils@2.0

Compilation context simply picks the latest versions, so : [mylib.d.ts,myutils@2.d.ts]

Hacky

Note that if you are using the browser (single version of each js) then this is probably what you want.
If you are using Node then even though example 2 will work fine we are forcing myutils@2 down the throat of mylib.

[poelstra]

@basarat Thanks for looking into this! 🌹

In my mail I saw a different post than the one above, seems you've editted it :)

Anyway, you made a valid point, so let me comment on that:

This gives a compiler error, and even if the compiler would somehow allow it,
would make it impossible to refer to either version of myutils.

myutils will (and should) not be usable in myprogram. So they can both have their declare module
"myutils" and if we don't read that in myprogram then myprogram should compile just fine.

I may have phrased it a little unclear. I agree that myutils should not be usable in myprogram directly.
What I meant was that the require("myutils") (or import ... from "myutils";) statements that will be present in mylib and myotherlib can only be resolved to a single declare module "myutils" {}.

Re your 'hacky solution': I don't think a conflict resolution like you're proposing is going to work, and may lead to uncompilable code.

Example of it going wrong using 'wrapped' external modules:

/// mylib/typings/myutils.d.ts (i.e. the 1.0 typings)
declare module "myutils" {
  export interface SomeOldInterface { ... }
  export declare function something(): SomeOldInterface;
}

/// mylib/dist/mylib.d.ts
declare module "mylib" {
  import myutils = require("myutils");
  export declare function myfunc(): myutils.SomeOldInterface;
}

/// myotherlib/typings/myutils.d.ts (i.e. the 2.0 typings)
declare module "myutils" {
  export interface SomeNewerInterface { ... }
  export declare function something(): SomeNewerInterface;
}

/// myotherlib/dist/myotherlib.d.ts
declare module "myotherlib" {
  import myutils = require("myutils");
  export declare function myotherfunc(): myutils.SomeNewInterface;
}

I didn't actually try to compile this, but I'm sure we'll get a compiler error on the return value of either myfunc() or myotherfunc() not existing (depending on the order of your resolution).

Contrast this with the following case:

/// mylib/typings/myutils.d.ts
export interface SomeOldInterface { ... }
export declare function something(): SomeOldInterface;

/// mylib/dist/mylib.d.ts
import myutils = require("myutils");
export declare function myfunc(): myutils.SomeOldInterface;

/// myotherlib/typings/myutils.d.ts
export interface SomeNewerInterface { ... }
export declare function something(): SomeNewerInterface;

/// myotherlib/dist/myotherlib.d.ts
import myutils = require("myutils");
export declare function myotherfunc(): myutils.SomeNewInterface;

Because external module resolution in mylib is independent of that of myotherlib, both will get their respective types.
Sure, it may still be 'hard' to e.g. directly refer to the myutils.SomeOldInterface type from within e.g. myprogram, because we don't have a direct name for it, so probably myutils should simply have 're-exposed' the type for convenience, but e.g. typeof stuff will work, and I can assign the result to any other 'compatible' interface.

[basarat]

In my mail I saw a different post than the one above, seems you've editted it :)

many times :)

[basarat]

Yes my hacky solution would fail miserably for that.

[basarat]

I've made your example concrete: https://github.com/basarat/typescript-node/tree/master/poelstra1/ts

Here is the directory tree :

If you open utils@1 / utils@2 / mylib / myotherlib will just work. The issue does appear at myprogram.ts. Would be great if we could come up with a spec

[poelstra]

@basarat You are (again) amazing 👍 🌹

Note though, that I think the structure you have created now resembles #2338 more than it does this issue (because myutils is a 'native' typescript package).

This is issue is more about the case when myutils is not a typescript package, and its typings are given in a typings/ folder in both mylib and myotherlib.

If you like, I can update your repo with two other folders (maybe clone them to poelstra2 and 3, to allow easy comparison):

• poelstra2 would reflect the 'unwrapped native module' case (i.e. a direct External module resolution logic #2338 test case)
• poelstra3 would reflect the 'unwrapped non-native module' case (i.e. my proposal for this issue)

[poelstra]

Alright, I went ahead, and created those two branches in basarat/typescript-node@22d703a

Obviously, they don't work yet.

• poelstra1 probably never will,
• poelstra2 needs External module resolution logic #2338,
• poelstra3 needs both External module resolution logic #2338 and something along the lines of this thread.

As I had to hand-edit the files/structure and have no way of verifying, there may be e.g. a typo in a path here and there, but I think it conveys the general idea pretty well.

[mhfrantz]

Can you extend the example (which I think is a good idea) to cover the case where the types come from DefinitelyTyped that has not yet been modified as you suggest?

One of the "features" of DefinitelyTyped today is the ability to have a single d.ts file that contains the declarations for many modules. The prime example is node.d.ts. How would that be handled?

Not all of the types are referenced with the appropriate module scoping. For example, in bluebird.d.ts, you can reference it and thus gain access to the Promise class. You can then use that type in your own d.ts file. I suppose the "right" thing to do would be to require the Bluebird module to be imported, and then only refer to the Promise class via the imported name.

I like the theory behind restructuring DefinitelyTyped as outlined here, but I wonder how long it would take to accomplish. I sometimes wish for a parallel repository like npm where we can simply publish the d.ts files by claiming the namespace for individual modules, rather than centrally managing it like the DefinitelyTyped repo.

Alternately, I want to be able to point to one or more alternate typings repos. If your proposal would allow extensibility in the list of repos, then that would allow new-style repos to be developed piecemeal.

[poelstra]

@mhfrantz You make some very interesting remarks!

• extending example with non-modified DT typings: good idea, I'll see what I can come up with
• many declarations in one file:
  • node.d.ts: this is a 'special' case, as Node's docs state that e.g. http always returns its built-in module. So I guess this is one of the few cases where declare module "..." {} still makes sense.
  • all other cases are probably possible once Generate a single declaration file for external modules #2568 is done
• bluebird.d.ts and 'directly' referencing Promise: actually, the way that typing is structured today, is causing us trouble every once in a while; I can call Promise.resolve() in any .ts file in the project, even if I forgot to import Promise = require("bluebird");, it compiles fine, but explodes at runtime. Of course, we could be using import bluebird = require("bluebird"), and use bluebird.resolve(), but we feel that is much less clear. So for us, not having that implicit Promise internal module would be a good thing. May be useful for the browser, but less so for the CommonJS (incl browserify) case, imho.
• restructuring DT:
  • it is not a requirement to restructure DT for this proposal to work, although you'd need to strip the declare module "..." { } off those typings 'by hand' (like we currently have to add it manually to some of them)
  • the existing typings would still work as well (I think, updated example should prove that later)
  • having 'decentralized' typings helps for some cases, but I like the 'quality constraints' that DT enforces before accepting typings. You'd probably get a large soup of low-quality typings if you let that go.
• alternate typings repos: my proposal enforces nothing about which typings repos to use. Basically, I assume that 'somehow' typings end up in a typings/ folder, with a certain naming convention. This 'somehow' can be through tsd (which already allows specifying other repos), you can manually place them there, create your own tsd-variant, etc.

Maybe, for the sake of reducing the scope of this issue, I shouldn't even have mentioned the suggestion to restructure DT. It might be that it happens automatically anyway, once #2338 gets traction.

[poelstra]

Concerning the node.d.ts typings:
It would be a pity if we had to 'manually' put a /// <reference> line at the top of each ts file, just to have node's modules. In a way, node.d.ts is very similar to lib.d.ts, being the ecosystem/environment/global context. Maybe a typings entry (list of filenames) in tsconfig.json would be nice, where all of these files would be 'preloaded' (like lib.d.ts is today). I'll let this sink in for a bit, and possibly create a separate issue for that later.

[poelstra]

@mhfrantz I just pushed basarat/typescript-node@467b8b0, which includes an unmodified Bluebird typing in poelstra3.

Additionally, it contains poelstra2poc and poelstra3poc folders, which are 'tweaked' proof-of-concepts:

• poelstra2poc is a fully working example (compiles and runs) of the two different versions of myutils in one program. It works because I 'manually' applied the name resolving to the require statements. It is basically what poelstra2 would do if External module resolution logic #2338 is implemented.
• poelstra3poc is a largely working example (compiles and shows correct type information on hover, but doesn't run). It does show the potential of having two different versions of myutils, and the fallback scenario to unmodified DefinitelyTyped typings. Note though, that it still won't allow e.g. two different (wrapped) Bluebird typings in the same project.

To test, simply fire up Atom Typescript, add poelstra2poc or poelstra3poc as Project Folder, and open e.g. myprogram.ts. Hover over variables.

[poelstra]

@mhfrantz Applied the suggestion you had to rename the bluebird import from Promise to BluebirdPromise. Thanks!

Interesting to note that the hover over the myAsync() call in myprogram.ts shows Promise, instead of BluebirdPromise, even though the latter is present in myotherlib.d.ts.

[MicahZoltu]

Wouldn't this be solved by having #2338 support multiple typing definition files (whatever they end up being called)?

• myprogram
  • main.d.ts
    • contains import { mylib } from 'mylib';
    • contains import { myotherlib } from 'myotherlib';
    • contains import { myutils } from 'myutils';
  • tsconfig.json
    • contains "typings": { "myutils" : "myutils@3.0/myutils.d.ts" }
    • contains "typings": { "mylib" : "mylib@1.0/main.d.ts" }
    • contains "typings": { "myotherlib" : "myotherlib@1.0/main.d.ts" }
  • myutils@3.0
    • myutils.d.ts
      • contains export myutils
  • mylib@1.0
    • main.d.ts
      • contains import { myutils } from 'myutils';
      • contains export mylib
    • tsconfig.json
      • contains "typings": { "myutils" : "myutils@1.0/myutils.d.ts" }
    • myutils@1.0
      • myutils.d.ts
        • contains export myutils
  • myotherlib@1.0
    • main.d.ts
      • contains import { myutils } from 'myutils';
      • contains export myotherlib
    • tsconfig.json
      • contains "typings": { "myutils" : "myutils@2.0/myutils.d.ts" }
    • myutils@2.0
      • myutils.d.ts
        • contains export myutils

When TypeScript is parsing myprogram/myotherlib@1.0/main.d.ts it would walk the directory tree up looking for the first tsconfig.json that has a a matching (including * matches) typing key. It would find a match at myprogram/myotherlib@1.0/tsconfig.json and use that to figure out how to locate myutils.

This would solve the problem of packages shipping with their dependencies included inside of them.

[MicahZoltu]

There is, of course, another scenario where your dependency manager doesn't nest like this (to limit duplication). Take this scenario (imagine all the same file contents as above except for the tsconfig.json files):

• myprogram
  • tsconfig.json
    • "typings": { "myutils" : "dependencies/myutils@3.0/myutils.d.ts" }
    • "typings": { "mylib" : "dependencies/mylib@1.0/main.d.ts" }
    • "typings": { "myotherlib" : "dependencies/myotherlib@1.0/main.d.ts" }
  • dependencies
    • myutils@1.0
      • myutils.d.ts
    • myutils@2.0
      • myutils.d.ts
    • myutils@3.0
      • myutils.d.ts
    • mylib@1.0
      • main.d.ts
    • myotherlib@1.0
      • main.d.ts
  • source
    • main.d.ts

If left like that, all of the libraries would resolve myutils to dependencies/myutils@3.0/myutils.d.ts. This is unacceptable, so the dependency manager would need to add a tsconfig.json to each of the dependencies that points at the correct version. Here is what it would look like after the dependency manager added the appropriate tsconfig.json files:

• myprogram
  • tsconfig.json
    • "typings": { "myutils" : "dependencies/myutils@3.0/myutils.d.ts" }
    • "typings": { "mylib" : "dependencies/mylib@1.0/main.d.ts" }
    • "typings": { "myotherlib" : "dependencies/myotherlib@1.0/main.d.ts" }
  • dependencies
    • myutils@1.0
      • myutils.d.ts
    • myutils@2.0
      • myutils.d.ts
    • myutils@3.0
      • myutils.d.ts
    • mylib@1.0
      • main.d.ts
      • tsconfig.json
        • "typings": { "myutils" : "dependencies/myutils@1.0/myutils.d.ts" }
    • myotherlib@1.0
      • main.d.ts
      • tsconfig.json
        • "typings": { "myutils" : "dependencies/myutils@2.0/myutils.d.ts" }
  • source
    • main.d.ts

I believe this is a more realistic (and more ideal) example than the previous one, but it requires more tooling buy-in. Personally, I think tooling buy-in will come naturally once you give a well defined way to solve this sort of problem. TSD, JSPM, etc. can all easily generate the appropriate configuration files when installing dependencies. In fact, JSPM already creates a couple for similar reasons.

[poelstra]

@Zoltu Thanks for chiming in!

Wouldn't this be solved by having #2338 support multiple typing definition files (whatever they
end up being called)?

Do I understand you correctly, that you're suggesting to use a lookup in a typings key in tsconfig.json, instead of using the filesystem for it? And 'manually' specifying which version of a typing to use in that typings key?

If so, I think there's still two different cases:

1. the package is a TS package (mylib and myotherlib, in my examples)
2. the package is a non-TS package (myutils)

For 1, I think there's no need to explicitly specify anything, it should automatically work through the resolution magic in #2338.
For 2, would it make sense to derive that version from e.g. node_modules/myutils/package.json instead?

That would ensure that the correct typings version is always chosen, instead of having to remember to update tsconfig.json after e.g. npm update. Maybe semver version globbing could be used to match .d.ts versions to package versions then. Still, the 'search path' needs to be either hardcoded (e.g. typings/), or configurable (through tsconfig.json?).

[poelstra]

There is, of course, another scenario where your dependency manager doesn't nest like this
(to limit duplication).

I have a feeling that you're trying to 'redirect' all requests for a certain version of a certain package to one single .d.ts file, correct? So if two TS packages both use e.g. Bluebird (or at least the same major+minor of it), that there's really just 1 .d.ts file referenced?

I've been thinking about that before, but I imagined this could work by creating symlinks (even on Windows). So it would simply work with 'normal' files, but a tool like tsd could be enhanced to not create 'real' files but symlink to a central location somewhere.

In that case, that tool could also automatically deduce the version by looking at e.g. node_modules/myutils/package.json's version, then look on e.g. DefinitelyTyped for the corresponding version, download it to some local 'typings pool', and create a symlink from typings/myutils to that pool.

Note that de-duplication is only an 'optimization', not a strict requirement, because e.g. the same interface definition in two different d.ts files will be equivalent.

We used to do some bower in the past, but now switched everything to npm, so I may very well be overlooking important use-cases that you may have.

[MicahZoltu]

I apologize for not being clear, I don't think that a single d.ts should be used for different versions of a package.

What I am proposing is that keeping typings and JS libraries in sync is not the job for TSC, it is the job for package managers or other tooling. The same goes for keeping tsconfig.json up to date with the correct paths. The first step is to have the compiler accept a well defined tsconfig.json to resolve typings, then tooling authors (JSPM, NPM, Bower, etc.) can add support for TypeScript.

What I envision is the following:

1. npm install --save-dev jquery-typescript-definitions
   • Side Effect: jquery-typescript-definitions depends on a particular version of jquery and both are installed to my dependencies with well defined version numbers.
   • Side Effect: jquery is added to my tsconfig.json typings map as a key with a value of node_modules\jquery-typescript-definitions@1.2.3\jquery-typescript-definitions.d.ts.
2. import jquery from 'jquery';
3. tsc
   • node_modules\jquery-typescript-definitions@1.2.3\jquery-typescript-definitions.d.ts would be used for type information for jquery at compile time. At runtime, default resolution would occur for whatever that import compiled down to (depends on your module settings in tsconfig.json)

In the above example, you are downloading a jquery definition package (from definitely typed or something) that is mapped to a very specific version of jquery (as a dependency). This way, the author of the definition can tell you exactly which version of jquery they are defining and you don't have to try to figure it out on your own or deal with problems that come up because your definitions don't match the underlying code.

The same steps would be followed if you downloaded a package that came with its own definitions. The only thing that would change is that the package would likely not contain the -typescript-definitions suffix and the tsconfig.json typings map would point to the same NPM package as the runtime would (just different suffixes).

In the case of nested dependencies, see what I have above but remember that any definition package in the dependency tree can depend on a regular JavaScript (no definitions) package that contains the JS for which the definition package defines.

If you liked that one, you may also like:

How much wood could a woodchuck chuck if a woodchuck could chuck wood....

[MicahZoltu]

@poelstra To directly reply to your first question: I do think that there should be an explicit mapping for everything. While auto-mapping is nice for hello world applications and small scripts, I would much rather have explicit control over how things get mapped.

That being said, I would definitely not want to manually fill out these mappings by hand all the time. I believe tooling can easily solve this problem though. In modern languages we find it insane to manually manage dependencies (at least we should) like back in the C/C++ days. The same should be true about manually managing project configuration files. This is part of what makes NuGet for C# so great, the tooling does all of the heavy lifting "magic" to make things "just work" but the compiler doesn't need to know anything about where the packages came from.

TL;DR: The compiler should not contain auto-resolution logic or magic file resolution. If someone wants these things tooling can be built separate from the compiler such as into the package manager.

[poelstra]

npm install --save-dev jquery-typescript-definitions
... depends on a particular version of jquery

That means jquery is installed 'below' jquery-typescript-definitions, so I still can't require() it in my package directly.

Side Effect: jquery is added to my tsconfig.json typings map

That requires some very TS-specific changes to npm, doesn't it? I think it's 'easier' to get TS to adapt to NPM than NPM to TS.

This way, the author of the definition can tell you exactly which version of jquery they are defining

That's a very good point, and I initially thought this would be a good idea too. However, we're then depending on the author of jquery-typescript-definitions to keep up-to-date on newer versions of jquery. In practice, though, after a year of TS development we've become a bit reluctant to this type of dependency (e.g. gulp-* packages lagging behind the package they're wrapping). And we've also noticed that a package's typings generally keep working surprisingly well, as long as the major version doesn't change.

I do think that there should be an explicit mapping for everything

I tend to agree, having less automagic is always nice.

But I don't really see why a list of 'package name -> filename' in a typings property in tsconfig.json is fundamentally different from e.g. a typings/<package name>/index.d.ts-style lookup. Of course, we could discuss whether the path(s) to search in should be configurable in tsconfig.json.

One of the main reasons why I prefer to use the filesystem instead of an entry in tsconfig.json, is that tsconfig.json needs to be in version control (for the compiler and formatting options), and having tooling automatically update it is causing a lot of unnecessary merge conflicts etc.
(And now we seem to get an 'exclude' option, and maybe even globbing for the 'files' property, the only current source of merge-conflicts in tsconfig seems to be gone too.)

So, in my vision, your example would then go like:

1. npm install --save jquery
2. tsd install
   • Parses package.json, finds packages and their versions, downloads typings for these versions
   • Typings get saved as e.g. typings/jquery/index.d.ts or simply typings/jquery.d.ts
   • Optionally, tsd could decide to e.g. make that file or directory a symlink to a specific version of the typing in a 'global' typings cache folder (e.g. typings_cache/jquery/1.2.3/index.d.ts)
3. import jquery from 'jquery';
4. tsc
   • typings/jquery.d.ts would be used for type information for jquery at compile time. At runtime, default resolution would occur for whatever that import compiled down to (depends on your module settings in tsconfig.json)

Benefits:

• No 'side-effects' necessary
• Explicit version management/checking (by tsd)
• Simple, explicit mapping to the definition file to use (i.e. typings/<package>.d.ts or typings/<package>/index.d.ts, no globbing or priorities etc)
• Mapping of typings to different locations can (by virtue of symlinks) be as complex as you want, but certainly doesn't have to be (i.e. minimal magic in compiler, complex scenario support by tooling)

[MicahZoltu]

However, we're then depending on the author of jquery-typescript-definitions to keep up-to-date on newer versions of jquery. In practice, though, after a year of TS development we've become a bit reluctant to this type of dependency (e.g. gulp-* packages lagging behind the package they're wrapping). And we've also noticed that a package's typings generally keep working surprisingly well, as long as the major version doesn't change.

Having things "just work" by allowing the user to run with mismatched js/d.ts is nice right up until you have a problem, then it is all of a sudden a huge problem instead of a simple one.

On numerous occasions I have wished Visual Studio didn't require exact matches for PDB files and source code. However, at least when I don't have an exact match I know what the problem is and the steps to fix it (either change my PDB or change my DLL). If Visual Studio made it all "just work" instead I would get incredibly bizarre debugging errors and oddities. The same is true for the nightmare that is C/C++ dependency management. You get a DLL/LIB and a header file. It will let you compile with a mismatch, but then you get sometimes incredibly difficult to debug problems. If instead it just told you at build time, "your stuff doesn't match" you have a relatively easy problem to solve.

That requires some very TS-specific changes to npm, doesn't it? I think it's 'easier' to get TS to adapt to NPM than NPM to TS.

Yes. I was using NPM in this example but I acknowledge that NPM changes for TypeScript support are unlikely unless TypeScript gets huge adoption. More likely though are other package managers like JSPM, which IMO is almost perfect for the browser development world. Alternatively, people could write wrappers around NPM that solve this problem or they could author pre-compile tools that do the work to map NPM-retrieved definitions to the correct NPM-retrieved libraries.

In your example, tsd is such a tool. Personally, I think tsd should also do the NPM fetch for you, as well as pulling down the typings.

That means jquery is installed 'below' jquery-typescript-definitions, so I still can't require() it in my package directly.

This is a very good point and a flaw in my most recent proposal, primarily due to the way NPM works (this is not a problem in JSPM for example). I suppose this would require revision to the proposal to support mapping the compiled code as well. An example structure off the top of my head:

typings: {
    jquery: {
        definitions: "node_modules/jquery-typescript-definitions@2.1.4/jquery.d.ts",
        source: "node_modules/jquery-typescript-definitions@2.1.4/jquery@21.14/jquery.js"
    }
}

The above would result in the definitions being found and also the compiler replacing the call to import jquery from 'jquery' with jquery = require('node_modules/jquery-typescript-definitions@2.1.4/jquery@21.14/jquery.js'). Again, this isn't a problem for SystemJS which has its own mappings, but for CommonJS module exports maybe this solution is the right choice?

---

I will be honest, I am not a node developer so I don't use NPM or CommonJS for my dependency manager or module loader. In the browser world, I believe JSPM/SystemJS to be the future of dependency management and module loading, and having a mapping file fits in nicely there.

I think part of the problem here is that TypeScript wants to work in two different worlds with two different paradigms. One of those worlds (NPM/NodeJS/CommonJS) are legacy systems that are stuck with choices made many years ago, without the benefit of hindsight.

My goal with my comments here and on other issues is to come up with a solution that is agnostic to the underlying dependency manager and module loader while being flexible enough to work with all of the existing systems.

I personally believe that the compiler should not depend on files being in a certain place on the file system (such as a typings folder). Instead, the developer should explicitly pass things into the compiler that the compiler needs, or provide them through a compiler config file (tsconfig.json). This way, the dependency manager and module loader can solve the problem in a way that makes sense for the environment and they (or other tools) can teach the compiler about their way of doing things.

[mhegazy]

Sorry for the delay, but we finally have the node resolution in place, and i believe we need to put this proposal into effect soon to avoid creating a reference hell for node packages.

I have a proposal in #4665, based on the issues and ideas in this issue, and would love to get your feedback.

[loucyx]

I just created a "Typescript NPM Package" and I get the following error:

Exported external package typings file cannot contain tripleslash references.

My package uses express, compression, mongoose and other npm packages. If I remove my /// <reference path="typings/typings.d.ts" /> then I lost the auto completion and I get "Cannot find module" errors. How do I make the .d.ts files available without using the tripleslash references?

[basarat]

How do I make the .d.ts files available without using the tripleslash references?

@lucasmciruzzi At the moment you need to document to the user that they need these external to your project .d.ts included in their project. List : node.d.ts express.d.ts ....so on

There are workarounds under discussion to handle such packages and packages that abuse (use) the global namespace

[loucyx]

Thanks @basarat! I'll keep an eye on this topics to update my package as soon as the solution is defined :)

[mhegazy]

This should be resolved with the move to @types packages and standardizing on npm for dependency management.

Please see my comment in #4673 (comment) for more details. also see:
blog announcement, discussion thread and handbook documentation for authoring declaration files