Support looking for modules under node_modules when importing

[chanon]

Update 5 November 2015

The functionality requested below is currently implemented in typescript since at least 1.8 with one main difference:

Instead of having typescript.main and typescript.definition properties, there is only one typings property which you can point to either a d.ts file or a normal .ts file.

If you're developing a module to just use locally, you can have the typings point to a .ts file, but if you plan to publish the module, it is recommended to have it point to a d.ts file. This is because you don't want your module consumers to recompile your module files, just consume its typings.

I have setup an example of using this here:
https://github.com/chanon/typescript_module_example

There is a documentation page here that has more information:
http://www.typescriptlang.org/docs/handbook/typings-for-npm-packages.html

Thank you TypeScript devs and all contributors.

Original issue / feature request follows

---

Motivation

In TypeScript it is a lot harder to re-use typescript modules compared to re-using npm modules in JavaScript.

It would be beneficial if the typescript compiler is smart enough to look in node_modules folders and package.json files.

The reason is so that npm module developers that use TypeScript might be able to start writing and distributing modules through npm itself. TypeScript would be able to piggyback on npm's infrastructure and wide support.

Example for node_modules

If we had:

./node_modules/concator/index.ts
./myApp.ts

And in index.ts we had:

export function concat(param1: string, param2:string): string {
      return param1 + ' ' + param2;
}

in myApp.ts:

import concator = require('concator');  // loads the module from node_modules
var result = concator.concat('I like', 'this.');
var wontWork = concator.concat('this will fail');  // compile error, concat needs 2 params

So basically, the compiler is smart enough to find the module in node_modules and it automatically uses the typescript version (index.ts).

Then when the code is compiled to JavaScript, it naturally uses the JavaScript version.

Importing Folders as Modules

A more basic case is supporting Node.js's popular rule of http://nodejs.org/api/modules.html#modules_folders_as_modules as suggested by @vvakame below and in the closed (semi-duplicate) #207 issue.

typescript.main in package.json

There could be an addition to package.json files to specify where the main .ts file of a TypeScript npm module is. This would be similar to the main key that specifies where the main JavaScript / .js file is but for TypeScript instead.

Eg package.json for an npm module named "myModule" located at node_modules/myModule/package.json

{
     "main": "./dist/index.js",
     "typescript": {
          "main": "./src/index.ts"
     }
}

In this example node_modules/myModule/src/index.ts would be the main TypeScript file and doing an import myModule = require("myModule"); would import the node_modules/myModule/src/index.ts file.

For a JavaScript coder writing var myModule = require("myModule"); in a JavaScript file, the require would load the node_modules/myModule/dist/index.js file as usual.

As can be seen in this example, the TypeScript src is in the node_modules/module-name/src folder and the compiled JS files would be in node_modules/module-name/dist.

Something like this could be a (semi) standard for TypeScript npm modules so that the TypeScript source is cleanly separated from the compiled JavaScript output.

A more basic case as suggested by @vvakame below is supporting the popular Node.js rule of http://nodejs.org/api/modules.html#modules_folders_as_module

typescript.definition in package.json

Another possible key for package.json for non-TypeScript (plain JavaScript) npm modules could be typescript.definition. This would point to a .d.ts file that defines the module for TypeScript users of the npm module.

So that an
import $ = require('jquery');
would automatically read a jquery.d.ts file defined in the typescript.definition key in jQuery's package.json and make $ the correct type.

Example format:

{
     "main": "./dist/index.js",
     "typescript": {
          "definition": "./index.d.ts"
     }
}

(This format is already used by tsd as @Bartvds explains below.)

Then us TypeScript coders would just have to try to get as many non-TypeScript npm module maintainers to merge our pull requests that have our .d.ts files and package.json typescript.definition keys.

If we succeed with that, then TypeScript coders' life would be bliss ... no more separate managing of DefinitelyTyped .d.ts files. Just npm install and you get your TypeScript definitions too! Automatically and up-to-date with the module version installed.

List of Benefits

What we get from all of this is

• npm modules can be written in TypeScript.
• Both TypeScript and JavaScript coders can use these modules
• The module users who use TypeScript get the benefits of static typing without the module coder (or user) having to use the usual method of writing (or generating) separate .d.ts files (so when the sourcecode of the module is already in TypeScript we don't have to have another .d.ts file to maintain)
• There would be a way to easily re-use and distribute TypeScript modules using npm which everyone is already familiar with
• It could help promote TypeScript usage itself, as JavaScript coders who use the npm modules written in TypeScript might see how nice / better the TypeScript source is and try switching to it. Or they might want to contribute to the module and since the source is in TypeScript they would have to learn it which may lead to them liking it and deciding to use it in their own projects.
• Basically it would help grow the TypeScript community through all the code sharing that could result. Right now everyone's TypeScript code is mostly their own / in their own silo. This is actually probably an extremely important thing as not having a proper/easy way to share/distribute/re-use modules is probably limiting the growth of the language. [1]
• Re-use of modules in internal projects would be a lot less of a hassle and would reduce the need for all those relative paths to .d.ts files and relative paths in module requires, and general .d.ts stuff. (Right now when writing TypeScript code, I find I have to learn to be good with counting ../../ s)
• This approach also supports Browserify/Webpack as Browserify/Webpack also look under node_modules, so this allows for easy reusable / npm distributed TypeScript modules for both the server and browser
• Non-TypeScript npm modules can easily be used in TypeScript with type information through the typescript.definition key addition. This allows .d.ts definition files to be packaged together with the npm module so that updating an npm module will automatically update its .d.ts definition file. This removes the need to update .d.ts files manually.
• Using definition files through the typescript.definition is simpler because it is simply an import moduleName = require("moduleName") statement with no need for a separate ///<reference ...
• Using definition files through the typescript.definition should also allow the use of different versions of the module in the same code base without type names clashing.

Detailed Proposal

@Nemo157 has written a very detailed proposal for how all of this should work at:

Proposed TypeScript Require Resolution Semantics
https://gist.github.com/Nemo157/f20064a282ee620f3877

An addition in the proposal is the usage of /typings folders which can hold definition files that can be automatically managed by tools such as tsd for JavaScript npm modules that won't include definition files in their repositories.

Final Supporting Facts

Since TypeScript compiles to JavaScript which runs mainly in two places: node.js and in browsers, supporting node_modules is beneficial to both places (practically all of where TypeScript is used) because of npm and Browserify/Webpack.

ie. there is no reason to come up with a different scheme when node_modules is what all TypeScript users use already for maybe 75%-100% of all their JavaScript code. (Taking out 25% for maybe RequireJS users.)

Footnotes

[1] - BTW I see there is a NuGet package manager (?) from Microsoft that can distribute typescript modules (?), but coming from a node.js focused (non .NET focused) background, I don't see NuGet becoming widely used outside of Microsoft focused shops, especially as npm is the standard for node.js and is a leading standard for client side JavaScript too. If I didn't use TypeScript I would have never heard of NuGet. The average node.js / client side JavaScript coder would probably prefer using npm, the same tool they use already rather than having to use something Microsoft specific such as NuGet. (I don't actually know a thing about NuGet so what I'm saying here might not actually matter.)

[chanon]

[moved to typescript.main in package.json in the issue description above]

[vvakame]

👍
I think http://nodejs.org/api/modules.html#modules_folders_as_modules is very popular rule of Node.js.

other example.

./test/index.ts

export function hello() { return "Hello, world"; }

./main.ts

import test = require("./test/");
console.log(test.hello()); // print "Hello, world"

[chanon]

[moved to typescript.definition in package.json in the issue description above]

[basarat]

One solution would be to solve it with better tooling. E.g. import foo = require('foo') gives you a hint to look for some local node_module+package.json (foo is a ts project) or DT definition (foo is a js project where library authors don't want to maintain the ts def).

[Bartvds]

FYI; I'm actually testing something similar to this in TSD; a way to expose & link definitions that are bundled in the npm (or bower) packages.

It's in my dev version for 0.6, and it works by adding a typescript element to the package.json (or bower.json), with a sub element definition (a sub element because maybe one day there'd be source too, or whatever).

{
    ...
    "main": "./index.js",
    "typescript": {
        "definition": "./foo.d.ts"
    }
    ...
},

Then you can run a command on TSD, currently tsd link and it will scan all package.json files in node_modules (or bower or whatever), find that property if defined and add a reference to it to the central tsd.d.ts bundle in your project.

Example: defined here and use here

[chanon]

@Bartvds That's pretty nice. Could be the first step for having .d.ts in npm packages. I like the package.json structure with "definition" in "typescript", it's a lot more organized.

If TypeScript compiler itself could read it automatically, would be very cool.

[RyanCavanaugh]

Tagging this for discussion -- smarter external module resolution is something we need to talk about to understand the various scenarios here. This was brought up previously and we made a few tweaks in 1.0.

[RyanCavanaugh]

Also discuss #207 - looking under 'index'

[Evgenus]

👍

@chanon tsMain is not necessary if we use declarations generation.

[wizzard0]

Yep, the issue is immensely important even for browsers - lots of projects use browserify/packify and therefore node-compatible directory layouts

[Nemo157]

👍

There's been some work already in this area back in the codeplex repo, one pull request by leebyron and one by kayahr.

I've been meaning to try and port one of them to the new repository, but it seems the related code has been re-arranged a lot.

[Nemo157]

I have written a proposal for how to add this to the TypeScript compiler. This also includes looking in the typings directory as that will be important when working with javascript modules that will not maintain their own typescript definitions. Unfortunately to make this all work transparently with tsd will still take a little work as the requirement of /// <referenced files being ambient external declarations makes things a little hairy. I will take a look at implementing this on a branch and providing some example modules using it.

[joewood]

Looks sensible. Will this cover modules with clashing type dependencies? I mean if an app imports A and B, and B is also dependent on A. It's easy to fall into duplicate type errors if there are two copies of the external declarations.

[Nemo157]

It shouldn't, they should resolve to different external module names so declare identically named independent types in different modules. The structural type checking should then allow the different modules to interact without issues.

That's one of the major issues with the current tsd definitions that this is trying to resolve, by defining external ambient modules there is no way for them to handle having multiple versions of a library with identically named, but slightly different types.

[Evgenus]

@joewood also there will be troubles between different versions of A

app
├── A@2.0.0
└── B@1.0.0
    └── A@1.0.0

[basarat]

there is no way for them to handle having multiple versions of a library with identically named, but slightly different types.

interested to hear solutions to the same. TypeScript has a global level variable scope for ambient declarations

[joewood]

Even if the versions match, at the moment I see issues with two different, but matching ambient declaration files causing duplicate symbol errors. The locally defined /// reference directive needs to somehow resolve back to a single file. Either that, or the type identity needs to include the path and structural typing will take care of version mismatches etc..

[Nemo157]

@joewood Yep, that's what I hope will be fixed in the majority of cases by changing require to load the correct definitions, avoiding /// reference wherever possible. The external modules will be identified by their absolute paths instead of having ambient declarations, so loading multiple of the same name from different paths and at different versions should be possible.

[basarat]

Yep, that's what I hope will be fixed in the majority of cases by changing require to load the correct definitions, avoiding /// reference wherever possible.

👍

[chanon]

👍 Thanks @Nemo157 and everyone for the discussion.

I've updated the top issue text to merge the package.json additions and linked to @Nemo157's proposal.

[johnnyreilly]

👍

[Nemo157]

I've done an initial implementation of my proposal along with creating an example set of modules to show that all the different ways to make a module will work. Including using different versions of the same library in different sub-modules.

[Lenne231]

👍

[bgrieder]

They can: const koa = require(‘koa’)

On 21 Jan 2016, at 14:47, Teoh Han Hui notifications@github.com wrote:

Why can't modules without type definitions be imported as such?

—
Reply to this email directly or view it on GitHub #247 (comment).

[teohhanhui]

@bgrieder I'm referring to the ES6 import syntax.

[bgrieder]

(that is is you are using commonjs), if not, use the declare var of Basarat answer

On 21 Jan 2016, at 14:48, Bruno Grieder bruno.grieder@gmail.com wrote:

They can: const koa = require(‘koa’)

On 21 Jan 2016, at 14:47, Teoh Han Hui <notifications@github.com mailto:notifications@github.com> wrote:

Why can't modules without type definitions be imported as such?

—
Reply to this email directly or view it on GitHub #247 (comment).

[joewood]

I think the point being made is that one of TypeScript's goals is that valid JS should be valid TS. For the ES6 module syntax, TS should compile this with or without the presence of type declarations. Without type declarations an import statement should just resolve to any.

I totally agree with that.

[bgrieder]

@joe Wood 👍

On 21 Jan 2016, at 14:52, Joe Wood notifications@github.com wrote:

I think the point being made is that one of TypeScript's goals is that valid JS should be valid TS. For the ES6 module syntax, TS should compile this with or without the presence of type declarations. Without type declarations an import statement should just resolve to any.

I totally agree with that.

—
Reply to this email directly or view it on GitHub #247 (comment).

[punmechanic]

FWIW this behaviour (expecting all 3rd party libraries to be typed ahead of time) is what is causing me to use Flow over TypeScript. While TypeScript has IDE support, expecting everything to be typed ahead of time isn't really reasonable, especially when there are plenty of libraries out there that:

• Aren't on DefinitelyTyped
• Don't have a ./typings folder
• I don't have the time to type it up myself

I'd rather not have to fight something that's meant to be aiding my development.

[mhegazy]

@danpantry, why not just declare your lib entry point as any, and the compiler will be content with that:

declare var $: any;

[joewood]

That's a workaround. But I think it would be a good idea for TypeScript to respect the ES6 module code and fallback gracefully. I can't see a good reason why an import statement can't fallback to any rather than reporting an error if the top level module is untyped.

[sccolbert]

I'd prefer the compiler to fail early and loudly (the current behavior), rather than allow runtime issues to creep in unnoticed. I'd prefer to explicitly have a declare var foo: any line to indicate "this is unsafe, and I know what I'm doing".

[bgrieder]

Why not just allow the ES6 import syntax when no typings are available an issue a simple compiler warning that the import has resolved to any, instead of an « emit preventing » error ?

On 21 Jan 2016, at 18:42, S. Chris Colbert notifications@github.com wrote:

I'd prefer the compiler to fail early and loudly (the current behavior), rather to allow runtime issues to creep in unnoticed. I'd prefer to explicitly have a declare var foo: any line to indicate "this is unsafe, and I know what I'm doing".

—
Reply to this email directly or view it on GitHub #247 (comment).

[teohhanhui]

In any case, the compiler currently claims that it can't find the module, when in actual fact it's just refusing to import the module without type definition. That's not really helpful, is it?

[jack-guy]

^^^ This a thousand times. There was no indication that I was doing something wrong when I first ran into this issue. It took some long hard Googling to finally figure out what was up (incidentally I think this issue is one of the only findable places that documents it). It's so unnecessary. And with Angular 2 bringing TypeScript more into the mainstream I can only imagine how many Stack Overflow questions will come from this behavior.

[amcdnl]

Very much agree with the poor error handling description and just make it import as any!

[punmechanic]

@sccolbert I agree there should be a notification but I'm not sure it should be a failure - a warning instead, perhap

[mark-buer]

I agree with @mhegazy and @sccolbert.

I'd much prefer our production build scripts (i.e. CI server) to complain loudly if something is amiss.

[devel-pa]

Hi guys,
This behavior just drives me crazy. Definition files are not up to date or not registered into package.json.
In the end TypeScript produces JavaScript, so, until we all go to this language pls be gentle to the rest of the world that provides libraries for the mother tongue in any other transpiled languages, like yours.
I really want to know if TypeScript wants to stay (I would say "to integrate" as it looks is not there, yet) in the JavaScript ecosystem or wants to live a life apart, as the second option will make me to go to something else.
To achive that a switch in the .tsconfig file will do, for that we can say that we want strict imports or not.
As for CI loud complaint there are test frameworks for that in JavaScript. Anyway, you'll not get type checking at runtime, the poor import checking is the less important issue to have.
All the best.

[mhegazy]

@devel-pa, the errors are meant to tell you that the compiler has no idea what your import is. any subsequent use of the import can not be checked.

Switching off the errors does not solve any issues. it just pushes these "unknowns" silently throughout your system. without type information, the compiler can not warn you on what is safe and what is not. and that is the whole point of TypeScript :)

As for generated JS. none of the TS type errors stop your output from being generated. if you do not care about the type errors, just ignore them all. the compiler still generates your matching .js files.

The solution for missing definitions is to declare them. you do not have to have the full shape of the module declared, but just the name. This allows the compiler to know that there is a module "myLibrary", it can warn you for typos in module names, and more importantly, no other modules would go unchecked.

declare module "myLibrary" {
    var a: any;
    export = a;
}

as outlined in #6615, TypeScript should support a shorter form of this soon.

[joewood]

I think the issue is pain this causes for onboarding projects. I think this problem is analogous to implicit any. Right now these imports are essentially implicit void. Sure, the errors can be ignored, but that causes a lot of noise and goes against the philosophy of gradual typing and valid JS is valid TS.

I can understand the confusion here for people new to TS. If they've actively used ES6 module syntax in JS, why doesn't it just work in TS? What's special about import from over var x = require - which is treated as an implicit any. Originally import was a TS specific keyword, so it implied that the developer wanted the module to be treated as a typed module. Now that it's not exclusive to TS I don't think that assumption can be made.

[mhegazy]

implicit-any errors are on variable declarations without a type. but using an undeclared variable is still an error today. For jquery, you still need declare var $; somewhere for the compiler to know that you really meant to have a global variable called $ and not just miss-typed it. it is more or less the same for modules, the compiler needs to know that there is a module called "myLibrary" and that is not a miss-typed name. so just add a declaration for it.

on any rate, #6615 should support adding declare module "*"; to match all modules in your system, though i think if you do that, you are not getting any help from your tooling, and you are setting yourself up for a painful transition later on when you decide to remove it.

[bgrieder]

@mhegazy I understand your reasoning but I really have a problem with this rationale

Basically, you are telling us to design a "placeholder" module definition, and the problem will go away.
This risk is that after a while, on a reasonably sized project, these "placeholder" definitions may just creep up and will be buried in some typings directory. Having no more warnings, everybody will just forget about them and when an issue arises, going through every module definition to see which is a placeholder, is just going to be a pain.

The situation is even worse with typed modules (the ones with the typings entry in package.json) that we would assume to be properly typed could actually be using those placeholders internally.

I understand these placeholders cannot be prevented, but I would much prefer that, at least, they are not encouraged.
What should be advocated is that by default, the import resolves to any and that the compiler issues a warning on every compilation. So at least, looking a the compiler/CI logs, we know something is potentially fishy and needs to be improved.

(as a side not, as stated on the typescriptlang.org home page, Typescript is a 'superset' of Javascript, and IMHO any valid ES6 syntax should just be swallowed as is)

[devel-pa]

Workarounds and hiding the garbage under the carpet are for me the worst that can happen (beside ES6 super).
I am working with JS libs all the time, and usually with last versions as part of my job and extra coding at home. 90% are not typed or have old defs, 9% are not very well typed (as the compiler doesn't know to make one def for all the files). Neither mine have very good defs, for the same previous reason and for the reason that my target is JS, I don't think I have to care about the original language.
Also, I've seen the reason that there are 'unknown' libs. No, not at all, if the developers doesn't know and understand what libs are used, why bother with them, I know why I'm using stuff and why I'm adding to the stack. Warning and tests (in case it exists) are enough for that.
Pls, JavaScript first, this is the target and the ecosystem. TS is just an accessory for better coding and type check at compile time, but only for what we are building. The rest is none of typescripts business.
I want to mention that I was against nmps peerDependencies too, I am the one who chose, not the software. Imperative programming, baby.

[phra]

what about importing modules from JSPM directory instead of node_modules?

edit: i've found an existing ticket -> #6012

[schmuli]

I just ran into an issue with Node module resolution, when using SystemJS to load a directory:

Apparently, while Node (and therefore TypeScript) understands that the path is actually a directory and therefore loads index.ts instead, SystemJS does no such thing, meaning that valid TS code will actually fail to load in the browser.

This is also true of node modules that have a index.ts/js entry point or even use the package.json main property. This is slightly easier to work with, because it is easy (although repetitive) to add a package configuration to SystemJS. This is not so easy when working with arbitrary directories.

What is the appropriate solution for this?

[mhegazy]

JSPM automatic module resolution is not currently supported. this is tracked by #6012.

The recommendation is to use path mapping module resolution support, see https://github.com/Microsoft/TypeScript-Handbook/blob/release-2.0/pages/Module%20Resolution.md#path-mapping

[netdragonboberb]

The above instructions are a start, but some people may need to do something additional... You may have to add
<Folder Include="node_modules" />
to .njsproj

More info

I was building with gulp, and had TypeScriptCompileBlocked in .nsproj. To fix issues in debugging breakpoints in VS 15 Preview 5 (I was getting "frame not in module" error) and issues I was having with intellisense I had to add to .njsproj . That directory was already there when I imported the project, but was set to git-ignore. That is my theory on why Visual Studio ignored it (perhaps it should have an automatic exception for node_modules?)

Beyond both debugging and intellisense working, I also stopped seeing intellisense errors beyond the same exact errors I saw from gulp while it built, which is expected.