Greatly improve code sharing and reuse options

[atrauzzi]

Creating and sharing pure typescript code has quite a few barriers that seem like the kind of complexity that TypeScript should be helping to alleviate.

I'm going through a situation right now where I want to share a pure typescript library and - unless I'm mistaken - it seems like I'm going to have to go through the pain of adding AMD to my project. Even despite the fact that it's all internal modules.

With the overall goal of making life easier for developers, I think it would be nice if:

• There was some way to reference typescript files statically such that they are used in the compile process. This could even be used to eliminate the /// syntax used to load .d.ts files.
• There was some way to tell typescript to prepend regular javascript files to the entire build output (dependencies with .d.ts definitions for example).

While AMD/CommonJS are module-loading systems, they are very confusing and cluttered build systems, especially once you bring TypeScript into the mix.

It's conceivable that with these changes, you'd also reap the ability to create build profiles implicitly by defining entrypoint scripts and having typescript crawl the dependency graph. Similar to how r.js works, except at a TypeScript level.

[atrauzzi]

It should probably be mentioned that any solution really needs to be sensitive to the inevitability that the location of the node_modules and typescript sources that the compiler is currently targeting will not be consistent between projects.

This might be one of the major pain points of coming up with a build system that knows how to integrate code from packages. Not only are people conflicted about npm vs bower, but what if what we really need are multiple node_modules directories in a project based on what's being built (front end, cli tools, server side)?

[danquirk]

For your first bullet, is tsconfig not a solution to the problem you're experiencing?

For the second bullet, this is something we've generally stayed out of. People have very varied build systems with a variety of linters, concatentation steps, minifiers, etc and it seems difficult for us to cover the majority of those scenarios in a satisfactory way (and convince someone to break their existing, working system to migrate to the provided TS one). Maybe the solution is to make it easier to customize pre and post build steps. We generally don't want to be in the business of creating some robust, general purpose, cross platform build system. Obviously it's important that the end to end experience be as good as possible though, so maybe we need to make changes to slot into existing workflows more easily or be better at detecting errors that might have occurred as a result of that system.

For node_modules #247 is a priority and hopefully helps alleviate some of the mismatch with using TS + npm modules now. Could you elaborate on what you mean by node_modules and TS sources not being consistent beyond that?

[atrauzzi]

I know I can't speak for others, but I think a build system integrated with typescript would be a welcome change to getting mired in endless build system hacking and configuration.

In an ideal world, I should be able to do this:

tsc path/to/built.js path/to/script.ts

BUT

TS does the legwork to resolve and include in the build everything that script.ts references.

What's preventing this from being a possibility?

[danquirk]

I think a build system integrated with typescript would be a welcome change to getting mired in endless build system hacking and configuration.

Sure, but this is really a complaint about the state of JS build systems and wishing for a great 'one build system to rule them all.' In an ideal world that would exist :) In reality peoples' needs are extremely varied which is why we have many different options (many tried to create the one true build system, now we just have a lot of different ones with different pros and cons). We don't feel like we're in any better situation to suddenly solve that problem better than all the other attempts (especially prior to ES6 having a finalized, cohesive module story). And then there's the fact that even if we actually did succeed at that, many people are not going to switch from their working build system to something new, even if the new thing is better in the abstract ('if it ain't broke...'). That said, we do want to try to slot into what exists well, and we do want to enable others to make good tools around us (like interacting with grunt/gulp/etc).

What's preventing this from being a possibility?

What do you mean by 'everything that script.ts references'? You mean a file without /// or imports? Or a file with ES6 style imports? Obviously right now the compiler does chase through your /// references and imports or will use a tsconfig related to that script. It sounds like you mostly want to avoid /// reference pain which is what tsconfig should help with.

[atrauzzi]

So for tsconfig, I have to manually create one for every graph of source files I want to use? Why not just have the compiler automatically infer this? r.js does this when you tell it to build a file. It will recurse through all the defines and requires and concatenate all those files together.

If I was able to statically reference other .ts files from one root-level .ts script, wouldn't that obivate the need for various mechanisms to tell the compiler about where files are? The compiler would know everything being used because it would have crawled the full import-graph (my root script, it's dependencies, their dependencies, and so on) before going through the code.

[danquirk]

So for tsconfig, I have to manually create one for every graph of source files I want to use?

Not for every graph necessarily, it's up to you whether you want to explicitly list everything or let reference chasing happen depending on what references are in relevant files. Think of it like a basic project file.

Why not just have the compiler automatically infer this? r.js does this when you tell it to build a file. It will recurse through all the defines and requires and concatenate all those files together.

This is exactly what the compiler currently does with .ts files based on the /// references and import/require in the file. With RequireJS you explicitly list your module's dependencies, and each dependency has its own dependency list, and the loader walks through this graph. That's the same thing tsc does when you explicitly list dependencies in the form of /// or import/require. The problem is that explicitly managing /// references in this way can be painful and sometimes things get pulled in in an unexpected order and now you have to manually try to track down how the graph was traversed and what happened. Using a tsconfig to manage the project ordering can alleviate some of this pain.

[atrauzzi]

So general advice for the time being is that I should be making everything external and not really try to make internal modules? That's a distinction I'm still trying to internalize, but I was under the impression that there might be a way for me to do all this without having to set up AMD....

[danquirk]

It's really up to you as far as how you want people to consume your code. Given that ES6 only has external modules it's certainly possible that's the right choice for you going forward though. The TypeScript compiler is only internal modules but you can find small npm packages that provide access to it through external modules. Likewise you'll see some .d.ts files written in such a way as to work with both internal and external (I believe knockout is one example like this).

[atrauzzi]

I'm still at a loss to really understand what the difference is between internal and external. Some of it flies over my head.

[danquirk]

I assume you've read https://github.com/Microsoft/TypeScript/wiki/Modules and http://www.typescriptlang.org/Handbook#modules ? Don't feel bad, it is unfortunately confusing and a situation we definitely can improve. I'm reticent to try to give an off the cuff primer since we've already spent a bunch of time trying to wordsmith the content in those two places since this confusion definitely occurs with some frequency.

The simplest distinction is that external modules are intended to be used with a module loader (either AMD or CommonJS) for all the reasons module loaders are good (plenty written on that topic for JS already). Think of internal modules simply as namespaces. Internal modules leave you with some JS (either many files or a single concatenated output files) that someone consumes by simply putting your file in scope (ex a <script> tag or concatenating with their own JS) without a separate module loader.

[atrauzzi]

So if I'm making the actual entrypoint script for a page, is it possible to author an internal module while also gaining the ability to consume javascript modules? Or am I required to be producing a module so that I can consume other modules?

[rotemdan]

Hi, here's the pattern I've been using for more than a year and a half now (for very large TS apps with more than a hundred source files). Though it may appear to stand in contrast to much of the design recommendations and advice you'll get from the TypeScript team, it has served me very well:

• Although a large part of my development is for Node.js, I use internal modules _only_ (I compile to a single file that works both on the web and in Node). I declare a single namespace/internal module, let's call it App.
• Every .ts file usually contains a single class and looks like this:

module App {
  export class SomeClass {
  ...
  }
}

• I compile all files to a single .js output file (external module preference set to "CommonJS" but I don't think it matters as I never declare any).
• I enable the compiled file to serve as a node module (when the node environment is detected) in this simple way:

module App {
  var runningInNodeJS = () => (typeof require == "function") && (typeof module == "object")

  if (runningInNodeJS())
    module.exports = App;
}

• I dynamically load external CommonJS modules (in this example the Node file system module), when needed, in this way:

import fs = require("fs");

module App {
  if (runningInNodeJS()) {
    var NodeFileSystem: typeof fs = require("fs");
  }
}

EDIT: The compiler will not emit anything for the import statement as fs is only use in a type position, see http://www.typescriptlang.org/Handbook#modules-optional-module-loading-and-other-advanced-loading-scenarios

EDIT: Removed solved problem: solution is in #2346 (comment)

I published a library using exactly this pattern:
https://github.com/rotemdan/lzutf8.js

Here's the compiled release file, I linked to the specific line where the internal module CommonJS export pattern is used:
https://github.com/rotemdan/lzutf8.js/blob/master/ReleaseBuild/lzutf8.js#L15

BTW I'm aware that the next version of TS will have ES6 modules but I haven't really looked deeply into them, though from what I've seen so far I don't feel I'd eventually make use of them.

[rotemdan]

As for build automation, I agree that there's no "one" optimal build system, but here's my own current workflow:

• All my compilation and builds (including compile-on-save) are now done through Grunt, which can be conveniently integrated into visual studio with the Task Runner Explorer VS extension: https://visualstudiogallery.msdn.microsoft.com/8e1b4368-4afb-467a-bc13-9650572db708
• Recently I even disabled VS initiated TypeScript compilation entirely, instructions can be found here: Visual Studio "No Compile" option #2294 (comment)
• In the Gruntfile I have a script (based on the Grunt API) that scans a set of glob paths for .ts files and generates a single references.ts file for the files found (it hasn't been published as a grunt plugin yet but may in the future):

function generateTypeScriptReferencesFile(filters, outFilename)
    {
        var fileNames = [];

        for (var i=0; i < filters.length; i++)
            fileNames = fileNames.concat(grunt.file.expand(filters[i]))

        function getPriorityByFileExtension(filePath)
        {
            if (/.+\.d.ts$/.test(filePath))
                return 0;
            else if (/.+\.ext.ts$/.test(filePath))
                return 1;
            else if (/.+\.spec.ts$/.test(filePath))
                return 3;
            else
                return 2;
        }

        fileNames.sort(function (fileName1, fileName2) { return getPriorityByFileExtension(fileName1) - getPriorityByFileExtension(fileName2) });
        fileContent = fileNames.reduce(function(result, filePath) { return result + "/// <reference path=\"" + filePath + "\"/>\n"; }, "");

        grunt.file.write(outFilename, fileContent);
        grunt.log.writeln(fileNames.join("\n"));
    }
...
generateTypeScriptReferencesFile(["**/*.ts"], "references.ts");

• The script also orders the files by priority according to their extension (.d.ts. and .ext.ts having the highest priority and .spec.ts the lowest, this scheme would probably be extended in the future).
• I compile the generated references.ts file with grunt-ts to a single outfile, where the debug output file will also contain Jasmine tests (originating from .spec.ts files) so the resulting file, having all its test suites integrated, can be tested both in the browser and in Node.
• Note that for a release build, for example, I simply don't include Tests/ and Benchmarks/ paths for the references file generator. In this way many different build configurations (debug, release, tests only etc.) can be made for a single project, simply based on inclusion and exclusion of directories. This cannot be currently done from within visual studio (though with the new tsconfig file it might).
• ..Other project specific build steps (such as minification, running unit tests, concatenation, adding banners, updating package versions etc.)..

An example Gruntfile.js (for LZ-UTF8) can be found here:
https://github.com/rotemdan/lzutf8.js/blob/master/Gruntfile.js

(Note this is a truly cross-platform solution and can also be run in Linux - I have it successfully building on a remote Linux container with Travis-CI)

Hope that was useful :)

[atrauzzi]

I guess I'll spend the next 5 days inventing my own build setup... 👎

[atrauzzi]

@danquirk - It seems like every explanation I read only gives a partial understanding, but I never get a full picture that I can put into practice.

For example: I'm trying to publish just some typescript interfaces as a package for two projects to communicate along. Literally no JavaScript will be produced (for now).

The top of my files read:

declare module MyProject {
    interface OneOfSeveral {
        ...

When I leave declare in, my .d.ts file contains the interface definitions. But I can't understand why when I remove declare, suddenly my .d.ts file is empty. Similarly, if I remove the module declaration all together, my interfaces are back in the .d.ts file. What is going on that's different here?

To compound the confusion, I'm normally used to structuring my files so that the namespaces are represented in a directory structure. There's a contradiction in my mind with explicitly naming modules and the way AMD works where the namespacing is usually just expressed through the filesystem. At which point, should I even bother namespacing any of my typescript code?

[atrauzzi]

So now I've also gone and split my interfaces into multiple files within a directory structure. tsc doesn't complain, and I have .d.ts files. But I have no clue how the interfaces that extend other interfaces are made aware of each other. Also, I worry that this is all happening at the root namespace in typescript.

[mhegazy]

@atrauzzi can you explain what you want to model? some concrete details would go a long way.

[atrauzzi]

I'm kind of past it at this point. I'm looking at JSPM right now, although I do hope that once TypeScript sorts out all these usability and code sharing issues, you guys will have an official plugin for JSPM.

I really love what you're trying to bring to the ecosystem, it's just not easy enough to use.

[atrauzzi]

Just going to add a reference here: #2233

As I mentioned, it would be great if there was a Microsoft-maintained plugin that built to ES6 and integrated nicely with the rest of the JSPM ecosystem.

@mhegazy : One of the issues is that the system is complex enough that I definitely throw up my hands trying to come up with an explanation. I'm not at all against a chat or email exchange with anyone who might want to tease out the sticking points. More than happy to be the guinea pig! :)

[mhegazy]

@atrauzzi there is a lot of abstract conversations going on in this thread, and i am finding it hard to get to the bottom of the issue(s) you are running into, or the ones causing you to feel that the system is too complex. If i understand correctly you are basically asking for the compiler to support node resolution process as described in issue #247. is this correct? if not it would be great if you can share some concrete information on what is it that you are trying to do and how the compiler is not helping, this way we can come up with specific fixes to simplify the workflow.

On a related note, @vladima is working on a proposal that handles some of the common module issues as specified in #2338, take a look and let us know if that would simplify your use case.

[atrauzzi]

@mhegazy

Regarding #247, I think that should be amended to support JSPM, which I think is the best JS workflow I've seen to date. To give you some some idea of the impact, I went from frustrated with TypeScript, to coding and running working ES6 in the span of a day. Whatever that project is doing, it was basically everything necessary to solve my issues.

Okay, with that established, I'll give this another shot and I'll try to isolate it in terms of what I go through in trying to use typescript. This is gonna be pretty dry! ;)

The initial authoring of my first typescript file brings up a lot of questions where I have to stop what I'm doing and research modules. When I see modules, I think namespaces and so my first attempt is to use them as such. This means that I'm going to create a hierarchy of directories containing files. What confuses me there though is that it looks like modules have no corollary in ES6. So I don't know if they're denoting a TypeScript design-time boundary, or a compiled-JavaScript boundary. Moreover, their purpose is ultimately unclear, given that there are explicit export keywords, similar to ES6. The behaviour of exporting when nested in a module declaration is odd.

The module/namespacing confusion gets more difficult when I'm trying to understand the impact it has on my ability to reuse code within my own project. When am I required to explicitly import, and when is an exported item from another file is available in my current scope? Because I have to consider both static and design time, I end up very confused. Even more befuddling is that there are two import mechanics. One is a straight up keyword, and the other is a frightening comment-hack. I just don't know when I'm safe to use something anymore because I don't know what keywords or constructs are going to require additional wiring from me at build time. Because TypeScript is your own ball game, why not just make the import keyword support importing .d.ts files, .ts files AND AMD/CommonJS names?

But now that exposes a huge huge huge point of confusion (I'm glad I typed it): What happens when you import a .ts file? Is the file concatenated to the top of the including script? Or is it simply added to the list of files to be built and added as an AMD/CommonJS reference? If it is added as an AMD/CommonJS reference, will the file be built in such a way that loaders can reference it (preserving folder hierarchy)? Very very very very confusing and unpredictable. Even if this is an "everybody is different" situation, TypeScript's documentation can't just gloss over the fact that what happens to translate imported functionality needs to be documented and ideally tutorialized. Right now though, people are kind of left with no guidance and frankly I will do everything possible to NOT have to configure a JavaScript build setup. I'll mention JSPM again here.

Continuing along this packaging+runtime thread, is the fact that if I want to publish interfaces to a package, I'm completely without a clue as to how to go about setting that up. I made a hamfisted attempt on Thursday to do this and just walked away from it. While I was able to define a hierarchy of namespaces full of interfaces, I had no way of producing a single .d.ts file (really that's all this library was going to produce) and then making a package that TypeScript would automatically become aware of once added to a project (maybe something JSPM can help with?). I was able to generate multiple .d.ts files, but that seemed impractical. I'd rather someone be able to just add one .d.ts file as a reference and get my whole library declared. But then would that introduce names into their file or would they have to then start importing things in cases where I wasn't just exporting interfaces? How would their code find JavaScript code in my package based on the TypeScript they import? Yikes.

So yes, I can author typescript files and pretend like I know what I'm doing. But when it comes time to compile, I feel like the compiler isn't clear on what it needs for one file to be able to resolve all its dependencies. I feel like it's unclear as to how the code I author is going to be called into at runtime, or even where it's going to live, both in my project and in packages I need to make.

I'm really hoping that you can glean something from this description. While I know it is mostly a summary of what I went through in this discussion, I just have no idea how to describe it any better. What might be nice is if you asked me specific questions based on all this.

What I can suggest is based on what I've seen from JSPM and how quickly I got up and running with it:

• You need to do something to make building TypeScript less or even no work for the developer.
• I'm not sure module declarations are useful given that modules are usually the entire file. They may be adding an additional layer of containment that have no counterpart in the runtime ES5/ES6 universe.
• I'd be willing to bet that if you got TypeScript integrated as a language option within JSPM, I'd be able to learn my way out of this confusion. The simple reason being that JSPM isn't just traceur/babel. It includes package importing, loading mechanics, ES6 transpiling and a full build process. I know that when I import a specific string-name in my ES6 file, it's gone and mapped that to a convention for each package.
• There needs to be a full and comprehensive guide on how to setup, author and build a TypeScript package. Again, ideally abiding by whatever is necessary to be loaded by JSPM.

It's all fairly abstract because this is as far as I've gotten. If at the end of the day this isn't enough, then the only thing I can say is: I have absolutely no idea how to set up my project to use TypeScript, there's just too much going on and there are too many sideffects and interactions that cross design, build and runtime boundaries to devise a mental model for it.

[mhegazy]

Thanks @atrauzzi for the details. i will need to read this again, but i got a few of concrete points, let me summarize some action items:

• Generate single .d.ts from external modules representing the package public surface area, instead of multiple (partially Suggestion: multi-file external modules #17)
• Better support load correct module sources (.ts) or definitions (.d.ts) based on the module system (External module resolution logic #2338)
• Add JSPM support (JSPM support #2233)
• Add documentation/walk through for creating, loading, publishing and reusing your first module
• Rename internal modules to namespaces to avoid confusion with external modules (Namespace keyword #2159)

Again i will need to read this again in the morning to get more details. Thanks for the feedback, really appreciate it.

[atrauzzi]

A quick addendum to namespacing corrections: Be sure to also go through it with a fine tooth comb and document their impact on design, build and runtime. There needs to be some emphasis on what they are to typescript and what they are to compiled code.

I'm sure you'll infer a few more details from my story above over time, but again - please ask me any time if you want to pick at a detail. I am still very interested in TypeScript and would love to see it as a language option just like traceur and babel in JSPM. That ecosystem really seems to "get it" and is coming together nicely.

[danielearwicker]

I think it's worth noting that TS as it stands does an excellent job of supporting the various ways in which JS is currently used. It's best understood as a tool for adding static type checking to the existing JS ecosystem(s), so it has to be flexible.

Some of @atrauzzi's questions are similar to questions I had when I started with TS and it was frustrating that they did not seem to be clearly answered anywhere. Now I get it, it seems like an elegant mapping to how JS is used in the real world.

But now that exposes a huge huge huge point of confusion (I'm glad I typed it): What happens when you import a .ts file? Is the file concatenated to the top of the including script? Or is it simply added to the list of files to be built and added as an AMD/CommonJS reference?

Neither. At the top of a module, when you say:

import x = require("x");

you are saying two things:

• You are telling the compiler that this is a module whose own declarations should not contribute to the global namespace. It may export specific things, but otherwise all its contents are private.

• You are telling the compiler to generate this in the output:

  var x = require("x");

(for CommonJS, or something conceptually equivalent if you choose AMD). You are also telling the compiler, only at compile time, to find type information for "x". It tries the following:

• Is there a declaration anywhere in the files passed to the compiler that says module "x" { ...? If so, assume that provides the type information for x.
• Is there a file called x.ts relative to this folder, or the parent folder, or any ancestor folder? If so, read it for the type information (but don't generate any code from it, as it is going to be passed to the compiler for separate compilation just like the present .ts file).

[atrauzzi]

So when distributing JS packages, I should also include a .ts file alongside every file so that TypeScript can "prefer" it when searching for dependencies? Is this documented behaviour?

[danielearwicker]

EDIT - this comment is no longer remotely true as of TS 2.0!

Going across package boundaries is totally different, unfortunately. TS doesn't know anything about npm (for example). When the consumer of an npm module says:

import x = require("x");

TS doesn't know that the implementation lives in ../../node_modules/x where there is a package.json file that might contain further helpful information.

The workaround is that the consumer must obtain from somewhere (e.g. DefinitelyTyped) a d.ts file that contains:

 declare module "x" { ... }

The consumer has to separately arrange to reference this file in their compilation step, which undermines the otherwise automatic nature and experience of npm packages.

But a package manage like jspm that supports transpiler plugins could make such assumptions. I've posted some vague wishes about this under issue #2233.

Such plugins take advantage of the host-ability of the TS compiler service, so they can add the necessary smarts to find type information automatically. This is the right way to do it, so TS itself is agnostic and doesn't become tied to specific ways of doing packages (new ways appear every week...)

[atrauzzi]

Fantastic, thank you so much for all this information :)

Hopefully it's enough to smooth the workflow so that I can switch over to TS at some point in the future!