Package typing inclusion and discovery

[weswigham]

This issue tracks our plans and discussion around the current situation with package typings.

Internally, we've been talking about how to better package, locate, and discover typings for packages which do not include their own typings.

The issues which arise around packages and their typings include:

• Reuse - the ability to take typings written for long-ago TS (ambient declarations) and continue to use them with little or no modification
• Dependency management - typings typically depend on the same things that their host js packages do - and usually on their typings, not their js.
• De-duplication and flattening - Deeply nested typings (with potentially duplicated dependencies at multiple levels) lead to a lot of duplicated work in the compiler, and could result in longer build times. We'd like to head this off before it becomes an issue in practice if possible.
• Discovery - if I'm using package the node package bluebird, how can I find types for it without skipping a beat?

We've spoken with @blakeembrey and his plans for typings (a tool to replace/improve upon tsd). We believe that we're both setting out to solve a lot of the same problems, and we'd love to help build much of that functionality into the compiler and language service to create a seamless development experience.

We've come up with a number of improvements to pursue on our end, including:

• Package scopes (Package scopes #4913) to help with typing reuse and isolation issues
• Extending our node module resolution for typings to have a concept of a fallback-typings package - a package which provides typings for a correspondingly named package. (For example, typed-debug can be installed and provides types for the debug module.)
• Reducing the volume of type checking done for packages (ie, don't typecheck them)

We want to make it seamless for users to use typescript with their existing tools and processes, and this includes their dealings with typings. Ideally, we want users to be able to npm install their dependencies as they already do and have everything "just work" without further steps or instructions.

Tagging @mhegazy @ahejlsberg @vladima @DanielRosenwasser and probably many others who should get tagged for discussion.

I'm sure I've forgetting things from our discussion earlier, so feel free to chime in with the relevant conversion points I've missed.

[weswigham]

Cross-linking typings/meta #3 for the other side of this discussion.

[jasonswearingen]

I just came here to file a related bug, but I suppose I'll post my anecdote in this discussion instead.

Anecdote

I have creating a monolithic library and am trying to publish it on NPM so others could use: https://www.npmjs.com/package/xlib

With Typescript 1.6 I thought I'd finally be able to have it be published and usable, but now that I'm attempting, I see the problem of d.ts collisions. My "xlib" uses typings for node.d.ts, so if "MyApp" imports xlib and also references node.d.ts on it's own, MyApp gets a hundred or so duplicate identifier collisions (defined in myapp/node_modules/xlib/typings/node/node.d.ts and myapp/typings/node/node.d.ts )

I was hoping to split my libraries into core/browser/node modules but because of the overlap (using browserify, plus MyApp needing it's own typings) this doesn't seem doable and I am going to need to move everything back into a single typescript project again.

Discussion

Ultimately I think the right approach (regardless of feasibility) is to have each project's dependencies be isolated, and non-interacting with other projects.

Where projects interact via imports (either in node_modules or via relative paths) there should be duck-typing analysis to dynamically and implicitly determine the type (shape) of the referenced code.

[blakeembrey]

@jasonswearingen You're absolutely right, which is why I've been working on typings. The only working story for this so far is typings, or you leave users to install the dependencies from your module. Obviously, the later is a bit of a pain in the ass.

As for the isolation, definitely. It's already working with typings in the current compiler versions (as far as I can tell, all versions). It's using a small trick to achieve this though. As for the node.d.ts use-case, I've commented before (in a few random places) but this is the use-case for "ambient" dependencies. I don't think environment dependencies should be installed alongside modules, as you're providing typings to things that may not exist at runtime can cause/mask subtle bugs. I'm currently leaving ambient dependencies as an exercise to the end user, but installation is simple and could catch things missing in the environment (think ES5 and Array.prototype.map).

[jasonswearingen]

Thanks for the details @blakeembrey It sounds as though typings holds some hope in the future.

Ambient Dependencies
I disagree with the idea of "ambient" dependencies though, as I assume it would add complexity and I feel it is very dangerous to implicitly type anything, as even node's api /behavior changes from time to time and best not to create a new sub-plane of DLL hell here.

My situation
It appears that typescript projects are really only usable (via NPM) if they don't reference any external d.ts definitions. For my xlib project I'll take a hard-dependency on a DefinitelyTyped repo sitting in a sibling directory until a better option is made public.

[blakeembrey]

@jasonswearingen

I disagree with the idea of "ambient" dependencies though, as I assume it would add complexity and I feel it is very dangerous to implicitly type anything, as even node's api /behavior changes from time to time and best not to create a new sub-plane of DLL hell here.

Could you elaborate on this? It sounds like agreement here. Ambient dependencies are not installed by default for that exact reason, the node API (or any ambient API) could change, so it's up to the user to have their environment APIs typed. This way, as a package developer, if you're relying on an environment API (e.g. streams1 -> streams2) the TypeScript compiler would error when the wrong streams API exists. It's less of an issue because things are slow moving, but it's even a built-in issue with TypeScript for now - the browser APIs are typed when all you're using is node and that can cause a subtle bug if your relying on type information that is inconsistent with the runtime API.

For my xlib project I'll take a hard-dependency on a DefinitelyTyped repo sitting in a sibling directory until a better option is made public.

Sure, but people consuming your package can't get any type information from that. And if they do have a dependency that duplicates from DT, you'll be creating pain for the consumer. Typings does currently work already and solves this. Though, I've only converted a few repos using it:

https://github.com/TypeStrong/tsconfig/blob/master/typings.json
https://github.com/TypeStrong/ts-node/blob/master/typings.json

Edit: In any case, there's still a few weeks of work before everyone can be using this. Especially for beginners to be introduced, since there isn't anywhere as many type definitions as DT right now. However, as you can see, for ambient dependencies I've been consuming them directly from DT anyway.

[jasonswearingen]

@blakeembrey I guess I'm misunderstanding something about Ambient dependencies, I am considering their use versus the ideal case of each module containing (and isolating) it's dependencies. In this situation a MyLib module might ship with a node.d.ts for node v0.8 and as long as an app consuming the module doesn't try to take some v0.8 specific output from MyLib and try to pass it back to a newer but incompatable version of node, it wouldn't raise any type errors. Of course maybe running the v0.8 specific MyLib might crash but I wouldn't expect any dev-time errors to be raised.

I do look forward to trying out typings once it's introduced to the public, I am still not sure how it solves my problems but I'm sure your future blog post will inform :)

[blakeembrey]

@jasonswearingen Ok, it's a very specific semantic I'm using right now. Let me try to tidy it up for you 😄 This certainly helps me, because it's helping me understand how I need to frame it and whether naming needs to change (ambient isn't the best name already, it's more like global or environment). Please suggest that change if you think it's clearer to have "global dependencies".

each module containing (and isolating) it's dependencies

Absolutely, each module has to isolate dependencies properly. However, node.d.ts is not a dependency. It's a definitely of the environment, which as a package author you have zero control over.

module might ship with a node.d.ts for node v0.8 and as long as an app consuming the module doesn't try to take some v0.8 specific output from MyLib and try to pass it back to a newer but incompatible version of node

This part is a little confusing to me. The environments have to be compatible, there's no choice as they will both be under the same node version at runtime. 99.99% of the time, you're unlikely to be using something unsupported across node versions, but in that one case I'd like to know my dependency on MyLib is going to break before it does in production.

Of course maybe running the v0.8 specific MyLib might crash but I wouldn't expect any dev-time errors to be raised.

Why? If there's something that will break, why wouldn't you want to know before you're invested in it?

Here's another example. child_process.spawnSync was added in node 0.12. If you author a library that uses it, and someone has the node 0.10 environment, they need to know that your module will break upfront - not when they hit that obscure method that is using child_process.spawnSync. However, as an author, you might have foresaw this (or they've logged an issue) so you quickly do npm install spawn-sync to "proxyfill/ponyfill/whatever people want to call it now". Then the type information is correct, not relying on the environment but spawn-sync module and everything runs smoothly without errors in both the compiler and runtime.

Does that example make it clear, or worse, on why I think these shouldn't be treated as direct package dependencies?

[jasonswearingen]

However, node.d.ts is not a dependency. It's a definitely of the environment, which as a package author you have zero control over.

Maybe node.d.ts isn't a good example, I'm just using it because many other d.ts files have a hard dependency on it so it's an obvious example of where project A will reference it and project B will too. For my examples (and how it currently behaves), it is a dependency that you install just like any other d.ts file. So maybe we should talk about something else people might use as a dependency... many different versions of bluebird.d.ts for example.

Why? If there's something that will break, why wouldn't you want to know before you're invested in it?

With my "it's just a d.ts file" argument, I don't think the typing compatibility really has any bearing on runtime compatibility. That's more for the package.json to deal with?

[blakeembrey]

I don't think the typing compatibility really has any bearing on runtime compatibility

Typing compatibility is runtime compatibility, unless I'm missing something. Yes, typing information is still decoupled, but what it describes is runtime compatibility.

So maybe we should talk about something else people might use as a dependency... many different versions of bluebird.d.ts for example.

Easy. That one isn't ambient, it's a normal dependency. Everything will work. Maybe you want to provide more information on what you mean here? Maybe exposing bluebird as the global.Promise, hence it's now an "ambient dependency". In this case, the compiler would do what it normally does - which I think is merge the declarations in this case (need to double check, but it's still up to the user/compiler here).

Edit: I just realised you're probably looking at this in terms of DefinitelyTyped. There's a reason I'm releasing typings apart from DT, and it's that the concepts currently aren't equivalent. In fact, the first version of typings was the rewrite for TSD (DefinitelyTyped/tsd#150). DT relies on the global to pass around type information, typings is designed to not do that. It's meant to be more realistic to the users runtime - for example, in DT any reference would be imported into the users project with any ambient definitions when they might not even be available. In typings, this will never happen.

[jasonswearingen]

Yes, I'm talking about a "DefinitelyTyped world", as that's the only one I have experience with!

Typing compatibility is runtime compatibility

In my DefinitelyTyped world they are certainly not the same, as the vast majority of d.ts files are incomplete or incorrect in various ways, and are rather constantly in flux.

[blakeembrey]

@jasonswearingen Ok, separate DT for a second and check out something like https://github.com/typings/typed-debug. You can see there's browser typings, and a dependency in typings.json on ms which is exposed in both the browser and node typings. It has the ability to replicate the same file structure, which means import 'debug/index' would work (I've had this issue with libraries, especially React and xtend which have "addons" or non imported exports).

In my DefinitelyTyped world they are certainly not the same, as the vast majority of d.ts files are incomplete or incorrect in various ways

Right, but I'm pretty sure they are meant to indicate compatibility 😉 That's an issue with DT maintenance and why typings has been structured so differently. Every typing can live anywhere, even in their own repos, which allows maintainers to properly follow only the typings they have committed to maintaining. This also allows anyone to write definitions and use them, without needing them to exist in DefinitelyTyped first (less friction).

I'll probably leave it at this for now, until I get a blog post done, but you've given me plenty of content and things I need to clarify 😄

Edit: Unless you have more questions, of course!

[jasonswearingen]

I'm pretty sure they are meant to indicate compatibility

From experience this is usually not the case, but rather just missing optional fields or non-critical return values incorrectly set as void

For example I am right now following a gulp + browserify tutorial and see that the DefinitelyTyped browserify.d.ts constructor options interface is missing many optional parameters

interface Browserify {
  (): BrowserifyObject;
  (files:string[]): BrowserifyObject;
  (opts:{
    entries?: string[];
    noParse?: string[];
  }): BrowserifyObject;
}

[blakeembrey]

@jasonswearingen What you just explained is the fact that it's not compatible, but it's should to be, which is exactly what I said and you quoted. I'm not sure what went missing in translation there.

[jasonswearingen]

sorry, I am talking about the a program using typings from a mylib.d.ts being 100% runtime compatible, even though the typings are slightly different. That's my example of where "Typing compatibility is not equal to runtime comparability" The typings may not be compatible due to an oversight in the typing, though they are still runtime compatible. I have seen this many times, for example when I wrote the hapi.d.ts definition it was very hard to convert the documentation into a d.ts file and as such there were (are!) many small typing discrepancies.

[blakeembrey]

@jasonswearingen You're still saying the same thing. You wrote a type definition, cool. It's describes the interface of a module. If it's wrong, that's a bug, nothing else. The environment doesn't even come into this. The type definition is describing a module.

I'm just not following what you're saying, but I feel like I've responded with same thing a few times so please try re-reading some of the previous responses and checking out the links I pasted. If there's a specific issue or input you have, please ask it.

Edit: Can you please open typing/packaging issues in https://github.com/typings/meta. I don't want to continue derailing this issue. The initial issue you posted is solved already with typings. It's the reason it exists, and I linked you to multiple places that explain this (including the original issue in TSD). The issue with DefinitelyTyped (and TSD), which was laid out in the first response, is that DT is based on ambient typings which cause version conflicts when distributed. Typings has solved that already. If that's the issue for you, I'll be sure to point it out in a blog post.

[jasonswearingen]

Sorry for increasing the frustration level! You are right that this is getting a bit rabbit-holed.

Back to directly discussing Package typing inclusion and discovery:

As per my original post in this thread stating my anecdote, Anybody here, please feel free to take a look at that xlib codebase if you want to see how someone (myself) is realistically trying to create a npm reusable typescript library. I had to abandon my attempt at npm modulizing / open sourcing xlib due to the issues described previously. If anyone wants more details as to my workaround, please let me know.

[blakeembrey]

@jasonswearingen I've been trying to explain it to you, so I'll try being as clear as possible. Using DefinitelyTyped, it's not possible. Using typings, it is possible. The goal here is to solve this, the issue is well know (I've published half a dozen TypeScript modules on NPM now). The only workaround (edit: when using TSD/DT), I know of, is to rely on the user installing the typings, which sucks and is incorrect (just like the references in DT).

Edit: If you have a typing problem and want to solve it, please open an issue on https://github.com/typings/meta where people can help you. Every package author that uses TypeScript has the same problem as you, including myself (hence typings).

Edit 2: You can also open this issue on https://github.com/DefinitelyTyped/tsd where I can also respond to you, if you're after help using DT - the right people will be listening there.

[dsebastien]

+1 for anything that will make the pain go away :)

I'm linking a set of related Angular 2 issues that shows how annoying things currently are and to help more people chime in: angular/angular#5459

[kitsonk]

And to pile on, we have long struggled with this for Dojo: dojo/dojo-package-template#3 and was hoping some sort of consensus would emerge and would obsolesce dts-generator.

[zpdDG4gta8XKpMCd]

@jasonswearingen consider using "peerDependencies" instead of "dependency" configuration property insider package.json of you xlib and app. This way you can avoid duplicates and redeclarations.

example: https://github.com/aleksey-bykov/generating/blob/master/src/package.json

[falsandtru]

Hi, I introduce my useful maintenance-free typings management implementation compatible with .d.ts files of DefinitelyTyped. I don't want to break this model by breaking changes of typescript.

mochajs/mocha#1933

[dsebastien]

The example of @falsandtru is interesting because it shows that some JS libraries maintainers are actually against including type definitions in their project. I've had the same experience with Firebase, I proposed the same thing and they did not seem interested.

Whatever the future of TS typings holds, it should provide a way to easily make the link between a typings file and some version of the library it describes the types for. Indeed it can still be out of sync, but at least there's some hope that it won't be completely so..

[jasonswearingen]

@Aleksey-Bykov Thanks for the detail, that indeed is helpful for my plans for xlib, though that's not the problem I was having in this thread (which is, multiple d.ts collisions) I might end up actually wrapping the various libs in a facade (thus normal "dependencies") if there isn't a complimentary d.ts answer to peerDependencies

[falsandtru]

TypeScript cannot discover own delivered typings of own and depended packages using package name when I use amd module. 😞

[mhegazy]

This should be resolved with the move to @types packages. see:
blog announcement, discussion thread and handbook documentation for authoring declaration files