[Feature Request] Rust-Like Macros that generate TypeScript (again)

[alshdavid]

🔍 Search Terms

macros, macro, rust

✅ Viability Checklist

• This wouldn't be a breaking change in existing TypeScript/JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
• This isn't a request to add a new utility type: https://github.com/microsoft/TypeScript/wiki/No-New-Utility-Types
• This feature would agree with the rest of our Design Goals: https://github.com/Microsoft/TypeScript/wiki/TypeScript-Design-Goals

⭐ Suggestion

I know this is an idea that has been beaten to death but I wanted to present a fresh case and slightly different approach for it as I do believe that there is substantial value we are missing out on.

---

TypeScript already has a macro; built-in support to convert JSX.

---

The suggestion being presented here is to generalize that idea to allow for new use cases and expand the ability for framework developers to utilize TypeScript without the need to create external compilers and LSPs/editor plugins (.vue .svelte, Angular's ngc).

Rust provides a well tested blueprint for a macro system that, on the surface, appears to fit well with TypeScript's design goals. That is; rather than expanding directly to JavaScript - macros expand to TypeScript which then is available to the type checker and LSP.

The simplest illustration of this is jsx. Jsx is a built-in macro included in TypeScript.

While not practical from a backwards compatibility standpoint, as an example, it would be possible to implement jsx as a callable macro.

From

/* compilerOptions: { jsx: "react", jsxFactory: "React.createElement" } */

const App = () => <div>Hello World</div>

To

/* compilerOptions: { macros: true } */
import macro { jsx } from 'jsxc' // stripped at compile time

const App = () => jsx!(<div>Hello World</div>)

I'm not fussed about the specifics of the syntax, just the functionality. In this proposal I will borrow Rust's syntax for illustrative purposes.

The use of the ! in jsx!() indicates that is it evaluated at compile time and expands into valid TypeScript.

The advantage of this is that the implementation of the macro would be externally maintained outside of the TypeScript core by the teams that utilize it (e.g. the React team).

Configuration (e.g. jsxFactory) would be targeted at the macro library rather than TypeScript's compilerOptions. This simplifies the objectives and configuration of TypeScript.

[expand macros] -> [typescript] -> [ecmascript]

But why?

In the front end world, we have many frameworks - but the only framework (forgive me for calling it that) that is practically supported by TypeScript is React and its derivatives. This is due to built-in support for expanding jsx syntax to its equivalent JavaScript syntax.

Other frameworks, like Angular, Vue, Svelte must all implement their own compilers, custom file formats and editor plugins (basically LSPs) to operate.

This increases the difficulty of innovating in the FE framework landscape, causing fragmentation, increasing the tooling burden for end users and preventing tool makers the ability to allow end users to update TypeScript beyond the version supported by their bespoke compiler (as is the case with Angular).

Some frameworks, like Vue and Svelte, are forced to use custom file formats, this causes difficulty when integrating with existing tooling (linters, formatters, test runners, nodejs).

How do macros solve these issue?

Formalizing compile time macros allows framework maintainers to use TypeScript directly rather than by using external bespoke compilers and stitching them together with editor/bundler plugins.

In addition, macros would expand into valid TypeScript at compile time, giving end users LSP warnings (which isn't possible with decorators) and type checking on the expanded syntax.

We see evidence of the success of this model in the Rust world; where there are many examples of GUI frameworks using no additional tooling and providing vastly different syntactic approaches to expressing GUI structures. Some resemble React, others resemble Vue.

• Web wasm frameworks [1] [2]
• Native desktop frameworks [1] [2]

FE Framework Examples

Compile time macros would allow a framework like Vue to use a macro for template compilation in its component declarations. This would remove the need for .vue files and the tooling associated to handle them.

import macro { vue } from 'vue-compiler' // stripped at compile time

export const HelloWorld = {
  tag: 'app-hello-world',
  template: vue!(<div>Hello {{value}})</div>),
  data: () => ({
    value: "World"
  })
}

Angular could do the same

import macro { ngTemplate, ngStyles, ngModel } from 'ngc' // stripped at compile time

export class HelloWorldComponent {
  static tag = 'app-hello-world'
  static template = ngTemplate!(<div>Hello {{ value }}</div>)
  static styles = ngStyles!(['./hello-world.component.css'])

  #[ngModel]
  value = 'world'
}

While the projects would still need bespoke compilers to parse/transform the contents of the code within their macros - that tooling would not be external and thus it would be natively supported by common tools (linters, formatters, test runners, etc).

It would also open the door to new types of frameworks - for instance Mobx-style observability could be more succinctly expressed with attribute macros. Something that currently requires painful amounts of runtime code and monkey patching.

#[observable]
class Foo {
  #[observe(push)]
  bar: string = ''
}

const foo = new Foo()
foo.observe(console.log)
foo.bar = 'updated'

Different types of macros

Take these examples with a grain of salt, I'm not the most creative person so these are more designed to illustrate how they work rather than examples of good macros that people would use.

Note: Only the macro engine is built into TypeScript, the macros themselves are implemented as user-land libraries that are installed via package manager or hand written

Callable macro

A callable macro is a function-like expression that accepts an arbitrary argument which it parses and returns valid TypeScript from at compile time.

Example of a macro that adds two numbers at compile time:

import macro { sum } from 'mathc'
const value: number = sum!(4 + 5)

Expands to:

const value: number = 9

Then transpiles to:

const value = 9

Note that the argument passed in is arbitrary.

A conventional sum function would take sum(4, 5) as two arguments but the macro takes sum!(4 + 5).

Behind the scenes, TypeScript would pass the internals of the macro call (you can think of everything within the sum!(<-->) as being supplied to the macro as a string) to the implementation, which then parses the input and computes the operation at compile time, replacing itself with valid TypeScript.

This means you can have a macro that accept jsx, html, csv, xml or some kind of custom language. It's language-ception.

Procedural macro

A procedural macro is an attribute for variables, classes and properties that augment the target, much like a decorator but at compile time.

Example of a macro that offers an "initializer" for a class and zero-values (making it struct-like):

import macro { struct } from 'structc'

#[struct]
class Foo {
  foo: string
}

const foo = Foo.new({
  foo: 'value'
})

Expanding to something like

class Foo {
  foo: string = ''

  static new(init: { foo: string }) {
    return Object.assign(new Foo(), init)
  }
}

Other Examples

Outside of the frontend framework world, many operations are needlessly done on the client. Having access to an API that allows for compile time computation means we have the potential to trade some loss in compile time for a speed up in runtime performance.

Macros could be written to embed external content into a file:

import macro { include } from 'macro-embed'
const FOO_TXT: Uint8Array = include!('./foo.txt')

A macro could be written to embed environment variables into the output

import macro { env, envFile } from 'maco-env'

const SOME_ENV_VAR: string = env!('SOME_ENV_VAR')
const env: Record<string, string> = envFile!('./production.env')

There are also use cases for macros where decorators were previously considered but ultimately unsuitable - like serializing/deserializing protobuf/arrow/etc and expressing database schemas as JavaScript objects and so on.

For brevity I will leave them out

Cons

Compile times may take a hit in projects that make heavy use of macros. This can be mitigated by leveraging wasm or napi modules within the macro implementations.

The general idea is that the complexity is handled by the macro maintainers while the end user has a simple api to work with.

Macros VS TypeScript's Design Goals

Examining how this fits into TypeScripts design goals:

• Statically identify constructs that are likely to be errors.
  • Macros expand to TypeScript which can be type checked and reported by the LSP
• Provide a structuring mechanism for larger pieces of code.
  • Reduces the amount of code needed to express something and reduces the tooling required to build a project
  • Rust-like macros don't share the limitations of other templating/macro systems
• Impose no runtime overhead on emitted programs.
  • Macros expand into TypeScript so there is no more runtime overhead than writing out the expanded forms of the macros by hand, and in some cases less runtime code would be emitted
• Emit clean, idiomatic, recognizable JavaScript code.
  • Because Macros expand to TypeScript which then needs to be type checked and compiled to JavaScript, there is no change here
• Produce a language that is composable and easy to reason about.
  • Beauty is in the eye of the beholder here. To me, Rust-style macros improve the ergonomics of the language, make it more versatile, easier to work with, potentially reduce the amount of tooling/configuration needed for a project, and potentially making it practical to use frameworks other than React/React-derivatives long term
• Align with current and future ECMAScript proposals.
  • In this regard, it simply adds a preprocessor system to TypeScript which does not affect the emitted code. You could write expanded macros by hand - it has no effect on the alignment with ECMAScript proposals
• Preserve runtime behavior of all JavaScript code.
  • It would generate valid TypeScript at compile time which then converts to valid JavaScript
• Avoid adding expression-level syntax.
  • I mean, not really? Doesn't add anything new to runtime code but it does add some syntax to indicate an operation occurs at compile time.
• Use a consistent, fully erasable, structural type system.
• Be a cross-platform development tool.
• Do not cause substantial breaking changes from TypeScript 1.0.

Implementation

Given the historically controversial nature of this topic, for now I will just say "look at Rust macros as a reference".

Comment Based

If TypeScript provided the ability to extend the LSP, it may be possible to implement this outside of the core as a preprocessor to TypeScript.

function App() {
  const value = "World"
  return /** @macro jsx!(<div>Hello{value}</div>) */
}

/** @macro #[struct] */
class Foo {
  bar: string
}

Mainline

If there is traction to incorporating this, I would be happy to expand on this and work on a reference implementation as long as there is a approval/desire to incorporate it into mainline TypeScript.

Forking TypeScript and producing a derivative that incorporates these features is undesirable as I feel it would be largely wasted effort.

[MartinJohns]

Duplicate of #4892. IMO opening a duplicate of an issue just because you don't like the decision is really not the way to good, especially when the definitive decision is only a year old.

[alshdavid]

It's slightly different to the original issue as it elaborates more on the concept and addresses some of the concerns of the original. Though I do understand the sentiment and apologise if it's inappropriate

[RyanCavanaugh]

We were aware of everything said in this issue the previous times we've decided not to do macros. I don't see anything new here.

The scope of this project is very well-defined and unfortunately I don't know how to articulate it any more clearly than we already have.

[alshdavid]

Fair enough, I'll close the ticket.

Part of the desire to include this into the TypeScript core is the lack of alternative pathways to add a feature like this.

I realize this opens the door to substantial fragmentation, but has the team considered enabling plugins for the compiler & LSP to facilitate the experimentation of these sorts of concepts?

I was thinking I could implement callable macros as jsdoc style comments as that's easy enough to work with in the existing compiler. However without LSP support that would be DOA.

function App() {
  const value = "World"
  return /** @macro jsx!(<div>Hello{value}</div>) */
}

/** @macro #[struct] */
class Foo {
  bar: string
}

[MartinJohns]

has the team considered enabling plugins for the compiler & LSP to facilitate the experimentation of these sorts of concepts?

#14419 / #54276