TypeScript extensibility

[billti]

TypeScript Extensibility

This is an umbrella issue for tracking the design of an extensibility model for TypeScript.

There have already been a number of issues opened with regards to supporting Angular2 via such a model, (especially for richer template editing support), so this will be the initial focus. This not only ensures rich support for a key TypeScript scenario, but by solving real problems, it also ensures we are not designing in a vacuum.

High level problems

This section captures an overview of the problems to be solved. These are not of equal priority, and not all features below may be implemented, but are captured here for completeness, and to be considered in the broader picture.

• Syntax highlighting: Template strings inside a TypeScript file currently show as simple string literals, yet ideally would be colored in a semantically meaninful way.
• Additional file-types and structure: Angular templates may live in a separate file, containing the HTML-like template contents. For React, TypeScript added a new file extension (.tsx) and matching grammar/parser changes for this file type. Ideally this would be an extensibility point, not baked into TypeScript.
• Syntactic and semantic diagnostics: Both the compiler and the language service should be able to surface domain specific errors in areas such as syntax, type usage, deprecation notices, etc. Such errors should map to original source locations, not intermediate artifacts.
• Rich IDE features: Statement and member completion, signature help, find all references, go to defintion, rename, etc. should work within the template contents as expected.
• Custom commands: Some plugins may provide editors additional functionality not exposed by the existing language service API (e.g. an editor visualization for the component hierarchy). A plugin should be able to expose additional commands for such usage.

Challenges

This section captures non-trivial challenges to be addressed, either with the problem domain in general or the current TypeScript architecture.

• Syntax highlighting: Several editors, (e.g. Sublime Text, VSCode, etc.), use a TextMate bundle for syntax highlighting. This is generally a static file describing the grammar for a file type, usually detected by file extension. Expanding the grammar via a plugin is a challenge. Even if statically augmenting the grammar, detecting when a string literal is an Angular template or not, or this .html file is an Angular template or regular .html file, is also challenging to do at parse time (and often depends on type information for inline template, or path resolution for external templates, to determine definitively).
• Syntax highlighting: TextMate aside, other editors (e.g. Visual Studio) do syntax highlighting via other means (such as calling the TypeScript classifier), which would need to be augmented somehow also.
• Program structure: In order to provide some of the above functionality, it is desirable to generate an intermediate representation (e.g. containing a "compiled template") and ask TypeScript questions about this code, then map the results back to the original source. However, the representation of the program that the compiler or language service has is immutable. It is not possible to receive a request about a source location, generate an intermediate representation of it, update the program with the generated code mid-request, then ask TypeScript questions about it before responding.
• Loading: TypeScript runs in environments besides Node.js (e.g. in-process in Visual Studio, or the tsc.exe command-line compiler), thus a plugin can not make assumptions such as being able to call require or take dependencies on other Node.js packages.
• Performance: As the program is immutable, a new program is created on every edit. Most language service operations require that the program is bound and type-checked, which is often done on-demand when a question is asked. For acceptable performance, (faster than 50ms response time), processing should be kept to a minimum and done entirely in memory where possible (e.g. avoid serializing/reading generated artifacts on disk such as temporary code, source map files, etc.).

Angular specific challenges

• Ambiguity: What if two different components have a templateUrl that resolves to the same file on disk? In which context should TypeScript evaluate that file?
• Determination: If a user opens an HTML file, how can the editor determine if this is an Angular template rather than just a regular HTML file without loading the full TypeScript program and resolving all the templateUrl properties?
• Path resolution: What if the path on disk doesn't match the path at runtime? Is some type of baseUrl or config object needed to map paths from the TypeScript source to the template files?
• Dynamic content: TypeScript needs to be able to statically analyze code that may be dynamically generated at runtime. For example, how to handle an inline template that reads <div>${getStart()}<foo [prop]='expr'/>${getEnd()}</div>, or analyze the component with the directives given as directives: getMyDirectives()?
• Name resolution: Within an expression in a template, scoping is quite different to normal. All instance members are available as top level names (i.e. no this. needed), and the usual global members aren't available (i.e. can't reference Date.now, parseInt, or similar). Thus resolving names and providing completion lists requires quite different logic to the usual TypeScript behavior.
• Micro-syntaxes: Within Angular templates, expressions are mostly JavaScript expressions, but not quite. For example, certain operators have different meanings, arguments are passed differently when using pipes, etc. See https://angular.io/docs/ts/latest/guide/template-syntax.html for more info.

Current work

The below is currently experimental work to spike various approaches.

• TextMate grammar to provide syntax highlighting for inline templates: https://github.com/billti/TypeScript-TmLanguage/tree/ngml
  • Note: This current uses a heuristic of determining if a template string is a value for an object literal property named template, or is preceded with a /** @html */ comment. See https://github.com/billti/TypeScript-TmLanguage/blob/ngml/TypeScript.YAML-tmLanguage#L507
• A fork of TypeScript with a rudimentary plugin model (plugin source lives within the codebase currently) and some initial feature support: https://github.com/billti/TypeScript/tree/ngml
  • TODO: Update with the latest TypeScript and Angular2 releases.
• Usage in VSCode:
  1. Replace the typescript.tmLanguage file under the install location at Contents/Resources/app/extensions/typescript/syntaxes with the version from the TypeScript-TmLanguage repo above (from the ngml branch).
  2. Fork the TypeScript repo above, checkout the ngml branch, and build.
  3. Open the VSCode global settings, and set the typescript.tsdk property to the build location in the prior step (e.g. /src/typescript/built/local).
  4. To get completions within the template on characters such as <, [, etc. open typescriptMain.ts from the Contents/Resources/app/extensions/typescript folder under the VSCode install location, and change the line that calls registerCompletionItemProvider to read vscode_1.languages.registerCompletionItemProvider(modeID, completionItemProvider, '.', '<', '(', '[', '\'');
  5. To make changes edit the local TypeScript repo, rebuild, and relaunch VSCode.
• Limitations:
  • Currently there is no dynamic plugin model. The plugin is compiled into the TypeScript service. To minimize build changes, all plugin code is lumped into one file currently (src/services/plugin-ngml.ts), as are some basic unit tests.
  • It currently uses its own rudimentary template parser. I haven't spent the time to figure out how to reuse the code from angular2/src/compiler (if possible)
  • All attribute values should be in single quotes. Unquoted or double quoted doesn't parse properly in the textMate grammar yet.
  • Interpolation inside attribute values is not wired up yet, i.e. <foo name='Hi {{name}}'></foo> won't work yet.
  • Attribute bindings (i.e. dotting off of attr) don't work yet, i.e. <td [attr.colspan]='expr'>
  • Only expressions where the first token is a member bind correctly, i.e. mem1(mem2) + mem3 will only bind mem1 to the class instance member correctly.
  • Components within components do not work yet (in fact, none of the directives values are resolved currently).
  • Special directives, such as *ngFor='#item of items', aren't implemented yet.
  • Angular specific syntax, such as pipes or the Elvis operator don't work yet, i.e. mem?.value | mypipe:"arg1"
  • Naming conversion still uses snake-case, and needs to be updated for the Beta release changes to attribute name mapping.
  • Rename isn't working yet (but is simple since the Beta change to simplify name mapping and is nearly done)
  • Still to figure out work for features such as outlets, etc.
• Current features
  • Completions and errors on HTML tags (using a basic list of HTML elements)
  • Completions on member expressions
  • Syntactic and type errors on member expressions
  • Signature help and tool-tips on expressions
  • Data and event binding using the [prop] or (event) syntax
  • Introduction of scoped template locals using the #name attribute syntax
  • Goto definition on identifers within template expressions
  • The below shows an example of the plugin in action

• TODO
  • Detail the architecture and which of the above challenges informed which choices
  • Figure out how to plug in to the compiler as well as the language service for the errors as build time as well as design time

Existing related issues:

• [typescript checking frontend template files #5151] typescript checking frontend template files
• [[tsserver] custom command (ex: to collect Angular2 @Component/selector) #5730] [tsserver] custom command (ex: to collect Angular2 @Component/selector)
• [Design Meeting Notes, 11/20/2015 #5740] Allow evaluation of a detached buffer in a specified context (angular2 templates)
• [Language Serivice support in Angular 2 inline templates #6477] [tsx] Angular 2 inline templates

[angelozerr]

@billti your demo is very impressive!

I tell me if your work could take care of HTML file too. You provide HTML, Angular completion for @Component/template, but what about @Component/templateUrl ? I mean:

• person.html

<h1[style. // here Ctrl+Space shows FontFamily

• person.ts:

@Component({
  templateUrl: " // here Ctrl+Space shows person.html
})

[billti]

This is partly the first bullet under challenges above, namely that we can't just assume any .html file is an Angular template, and you don't want to load the TypeScript language service and scan all source to see if it is anytime someone opens an HTML file. If it had a unique extension you could identify the grammar and language service based of this (e.g. *.ngml or similar).

With a unique extension that we know definitely is an Angular2 template, then editors could automatically apply the correct grammar (i.e. TextMate bundle in the case of Sublime or VSCode), and load the correct language service (and provide intellisense in the areas you outline above).

[angelozerr]

@billti if I have understood your comments, the challenge is to detect if an HTML file *.html is an angular template or not? For me this information is about the project nature. This nature could be detect :

• with an user settings (ex : for Eclipse you can set a nature in the .project)
• or could be detect:
  • search depdendencies in the package.json
  • or search if the project contains a node_modules/angular2 folder
  • or some other thing (ex : tsconfig.json, etc)

With *.ngml it fixes the problem with project nature, but not the full scan of typescript files to retrieve the well class which defines @Component/templateUrl

I'm a little afraid with this case for performance problem to have to scan the full typescript files. Or perhaps I have not understood something?

[ericmdantas]

That'd be awesome!

[billti]

@angelozerr That's half the battle. The challenge is more of a catch-22, and is something like "how do I detect if I need to load the language service, without loading the language service to do the detection logic".

For performance reasons many editors do not load the language service unless needed. Currently, an .html document never needs the TypeScript language service, thus it is rarely - if ever - loaded when opening an HTML document. Assuming the above logic would live in the TypeScript (or one of its plugin's) code, then you would need to load the TypeScript language service on every HTML document you open to detect if it is an Angular template. Does that make sense?

Maybe that's a trade off we'll have to make. It's just not one to take lightly.

[LPGhatguy]

@billti How much of the detection code needs to be baked into the full-fledged language service? Would it be possible to build a service whose sole purpose is to detect what a file is, but not do anything with it?

It could share source with the language service, but it would be a smaller process with an otherwise smaller source base.

[remojansen]

👍

[DanWahlin]

Seriously cool feature - looking forward to having something like that!

[angelozerr]

then you would need to load the TypeScript language service on every HTML document you open to detect if it is an Angular template. Does that make sense?

I had the similar problem with AngularJS Eclipse based on ternjs. The load of ternjs is done as soon as an HTML editor is opened but only if project contains a .tern-project (you could compare this file to tsconfig.json) which declare angular2 (you could do that by checking that package.json contains angular2)

I would like to know too if you could provide too custom tsserver commands to retrieve list of Angular2 components of a project to provide for instance:

• an Angular2 explorer like I had done for Angular1:

• a search engine to retrieve Angular2 component like I have done to retrieve module, controllers, directives for Angular1:

[mgechev]

@billti the last weekend I spend in prototyping a static code analyzer for Angular projects. It seems somehow related to the results you got.

I wrote a blog post about this, which could be found here. The prototype can be found at the ng2lint repository.

[billti]

@mgechev Nice! @chuckjaz has been investigating this tooling from the Angular implementation side, so he may be interested in some of the work you are doing here.

[chuckjaz]

@mgechev We are definitely covering some of the same ground. I read the blog post and I agree with @billti; Nice!. I will look at the code and see if there is some things we can share.

[chuckjaz]

@mgechev I read through the code and it seems the most obvious place for collaboration is metadata. We are working on a way to persist the metadata we need (since we need it for off-line template compilation) and we plan to reuse this information in the Angular 2 template language service. This is most helpful for classes for which you only have a .d.ts file. When producing a .d.ts file for a library you would be able to produce a metadata description for all the components and directives in the library. You could also use this persisted state in the metadata collector. We are working on this now but it will not show up anywhere for a few weeks.

You are already using the Angular 2 compiler so we are already collaborating there!

[mgechev]

@chuckjaz my current implementation uses LanguageService which discover definitions inside .d.ts files. At the moment I'm not persisting the metadata in the most appropriate format. I think it'll be better to extend DirectiveMetadata and component metadata because at this point directives is of type DirectiveMetadata[] and I need to do some dirty any-casting.

I'd love to continue the discussion on gitter, or hangouts!

[jods4]

Coming here from an Aurelia viewpoint...

One of the biggest issue in a large Aurelia SPA project today is that the bindings in your markup are not "part" of your TS code. So typos are not caught. References not found when you refactor something. And so on.
It would be awesome if TS could work seamlessly across html templates and .ts files.

Aurelia has some very easy default conventions. For instance, if you have a XCustomElement it has a view in file xCustomElement.html. Or if it has a decorator @useView("y") then it's in y.html... but then the problem is that some projects may choose to use custom conventions... Or some views are meant to be loaded dynamically and have no direct link with their ViewModel.

In those advanced scenarios it seems plain impossible to make it work seamlessly.
What if some metadata is required at the top of the file? asp.net or wpf-style... maybe:

<!-- aurelia vm=MyClass -->
<template></template>

Another problem is the concept of global resources. Those are "functionality" that you create and make available in all (or some) templates in your project. It is very hard if not impossible for a compiler to accurately analyze code so that it knows what resources are available in any given file. To give you an idea of the complexity: Rob has made a demo where he could change the syntax of the templates on a per-file basis in a single project, so that legacy Knockout or Angular 1 syntax could be used inside an Aurelia project.

Again it seems to me that to get this working properly one would need some kind of config file at the project level (maybe tsconfig.json extensions?).

[mgechev]

@MarkPieszak AFAIK @chuckjaz is working on this here.

[PaulBGD]

I'm guessing this is still planned for 2.1, but is there an active PR yet?

[angelozerr]

TypeScript 2.1 RC is now available. @chuckjaz has worked for Angular2 support but it works only for VSCode and it's not a tsserver extension. Do you think TypeScript 2.1 release will provide the capability to extends tsserver to support the work of @chuckjaz with Angular2 that in my case I would like to use to give Angular2 support for Eclipse.

[tomitrescak]

What about typescript compilation (emit) plugins, will this be possible to do with the plugin system? I'd love to see the possibility to change the compilation output and create plugins such as this one for babel https://www.npmjs.com/package/jsx-control-statements (HC React developers please do not throws stones, some actually like this :)

[squadwuschel]

are there some news for IntelliSense in HTML Files/Templates with TypeScript and perhaps Angular 2 for VS Code or Visual Studio?

[mgechev]

@squadwuschel here's a VSCode plugin for the Angular language service that @chuckjaz developed https://github.com/angular/vscode-ng-language-service.

[Quramy]

Hi. I'm inspired by @chuckjaz 's Angular Language Service plugin for vscode and I've create an extension, https://github.com/Quramy/ng-tsserver, for other editors which communicates with tsserver (Vim, Emacs, Sublime Text, etc...) .

[orta]

For people subscribed to this issue, there's been some merged work for TS 2.3 on this in #12231

Looking forwards to what we can do with Relay and TypeScript when this is 👍

[mhegazy]

Should be fixed by #12231. For more information about authoring a tsserver plugin, see https://github.com/Microsoft/TypeScript/wiki/Writing-a-Language-Service-Plugin.