How could we support typescript without vendoring it?

[mcollina]

I would like Node.js to support the following user experience

$ node script.ts
Typescript support is missing, install it with:
npm i --location=global typescript-node-core
$ npm i --location=global typescript-node-core
...
$ node script.ts
"Hello World"

(I picked the typescript-node-core name at random, I would be extremely happy if that could be ts-node)

Note that script.ts could evaluate to either cjs or mjs depending on the tsconfig.json file.

Originally posted by @mcollina in #43408 (comment)

---

Why vendoring is not an option?

1. TypeScript changes too fast and they do not follow semantic versioning, nor do they provide any long-term support.
2. We will never be able to pick a default tsconfig given the 100+ options.

[mcollina]

cc @cspotcode

[targos]

So basically you would like Node.js to automatically load something from a global predefined place when it starts?

[mcollina]

So basically you would like Node.js to automatically load something from a global predefined place when it starts?

Yes, but only when started with a .ts file.

[mscdex]

Wasn't that the main purpose of custom loaders? You should even be able to set that in a NODE_OPTIONS.

[GeoffreyBooth]

Please don't start issues related to loaders without tagging @nodejs/loaders

[mcollina]

This is not related to loaders. This is about the developer experience of Node.js.
Loaders are custom, user-specified components that are started before the app.
I'm talking about shipping something that would provide a better user experience for TypeScript users without additional configuration.

Please keep this issue about the user experience and not specific implementations.

[JakobJingleheimer]

I'm trying to see how this isn't reinventing the wheel, but I'm not getting it. Could you please explain how it's substantially different?

How I'm seeing it is: This is already easily achieved via Loaders with next to no effort (and that effort is basically set-and-forget). We have a simple working example of it in node/loaders-tests/typescript-loader. For a more robust one, ts-node provides such a loader (ts-node/esm). And what we currently have via loaders is super fast. We just switched to it at my work and saw like an 800% speed improvement.

PS In case we veer into the territory: I would vehemently object to TypeScript support in Node itself.

[mcollina]

I'm trying to see how this isn't reinventing the wheel, but I'm not getting it. Could you please explain how it's substantially different?

I would rather stay away from discussing a specific implementation of this. This could be loaders but it doesn't matter. I care about listening to our users. Once we agree on the user experience, then we figure out what's the best way to ship it. If it's already possible as you hinted, it would be terrific.

PS In case we veer into the territory: I would vehemently object to TypeScript support in Node itself.

That's what our users are asking for. We cannot provide TS directly into Node.js core for plenty of reasons (it's just not possible for the tsconfig mayhem), including the fact that the TS team think it's a bad idea. I propose that we implement the user experience described in #43818 (comment).

[aduh95]

A few questions come to mind:

• Are we making an exception for TS only, or are we open to adding more file extensions custom error message in the future?
• How do we chose what tool to recommend?
• Is Node.js core the correct place to solve that kind of problem?

Currently, what happens when someone runs node script.ts depends on the env:

• if we're inside a { "type": "module" } package, it will throw as .ts is not a recognized extension – in this case, it's quite easy to add a custom error instead as you're suggesting;
• otherwise, it will parse script.ts as a CJS module – in this case, we could try to parse the file as a CJS module, and if that fails display the error message you're suggesting?

We would then need a way to detect that typescript-node-core program is installed on the local env, and to defer to it to load that file. Maybe by making a $PATH lookup?

[GeoffreyBooth]

In an app with "type": "module" in its package.json, you get this:

node script.ts
node:internal/errors:477
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /private/tmp/test/script.ts
    at new NodeError (node:internal/errors:388:5)
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:80:11)
    at defaultGetFormat (node:internal/modules/esm/get_format:122:38)
    at defaultLoad (node:internal/modules/esm/load:21:20)
    at ESMLoader.load (node:internal/modules/esm/loader:431:26)
    at ESMLoader.moduleProvider (node:internal/modules/esm/loader:350:22)
    at new ModuleJob (node:internal/modules/esm/module_job:66:26)
    at #createModuleJob (node:internal/modules/esm/loader:369:17)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:328:34)
    at async Promise.all (index 0) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

So to provide the UX you’re describing, we would have to add a special case within this error flow where if the unknown extension is .ts, we print a special message. This will inevitably raise the question of what other unknown extensions do we want to print guides for; or we could print some message for all unknown extensions along the lines of “go to https://nodejs.org/guide-to-loading-non-javascript-file-types” and that documentation page could be a clearinghouse of instructions for various types.

In an app without "type": "module", however, script.ts is parsed and executed as CommonJS. If it happens to be runnable JavaScript, it runs. Otherwise it’ll error when V8 tries to parse TypeScript syntax. So to provide a similar experience in CommonJS, within that error path you’d have to add a check that the file that couldn’t be parsed had an unknown extension, and then print the message. Keep in mind that there’s a type annotations proposal that would make many TypeScript files into runnable JavaScript.

Either way, the solution that we would be recommending to users would involve adding the TypeScript loader. And transpiling TypeScript is squarely one of the use cases for loaders. So I object to treating this as “not related to loaders” and removing that label.

[aduh95]

@GeoffreyBooth let's imagine that the solution ends up looking like that (oversimplification):

if (entryPoint.endsWith('.ts')) {
  process.argv.unshift(entryPoint);
  entryPoint = '~/.node/typescript-node-core/bin.js';
}

Sure typescript-node-core will probably use loaders, but that's arguably an implementation detail. (FWIW I agree that a clean solution should involve loaders; if anyone is interested to work on this, that's where I would suggest them to explore)

[JakobJingleheimer]

What if it were some kind of…plugin (idk what the official term is relative to Node.js), like crypto or intl, with which node could be compiled or not? We could then have however many and we aren't responsible for them. It would be very similar to loaders, but a bit different and i think perhaps a little closer to what Matteo is talking about.

Also, this would avoid incorporating typescript-specifics (eg their .ts vs .js file extension nonsense) into node core (which is my main concern).

Annnnd it wouldn't require the CLI args everyone laments about loaders.

[avin-kavish]

Yeah, I think the important factor is user experience. Not sure how loaders work exactly but a DX like this would work - it could be specified in a configuration file which loader to use given an extension:

.m?ts --> use typescript loader
.cf --> use coffeescript loader
....  ---> any other user defined type

This could have global defaults and local (package.json) overrides. The global defaults would be shipped with node.js.

Maybe cli flags could be passed to the loaders, when needed. This way we can load any file conveniently with node path/to/file and spawned processes will also do so without additional config.

[cspotcode]

One thing to keep in mind:
I see in comments this is described as triggered by the entrypoint's file extension. There are situations where you may be running a .js entrypoint but still want a TS-enabled runtime, for example my-cli which executes my-cli.config.ts.

...but that's a detail.

High-level ask from users is still: "How do I opt-in to my runtime having automatic TS support?"

[cspotcode]

When we say that users are asking for TS support, is there a sense for the ratio of desire for TS support vs desire for <insert non-TS thing here>?

[benjamingr]

I am in favor of this and I think we are not doing a good enough job. I like Matteo's proposed idea of a blessed "how to run TypeScript module" - ideally I'd also like us to provide a node build or similar on top of it so people have a workflow for CI.

We'd probably need:

• To bikeshed/figure out the way to run TypeScript without extra flags in a blessed way (a hook and pre-installed package sounds reasonable)
• To figure out the story for passing flags (like transpileOnly) to TypeScript
• To figure out the production story (e.g. node build myProject) that will produce a .js file

Might also be interesting to know if this unlocks optimizations?

[benjamingr]

When we say that users are asking for TS support, is there a sense for the ratio of desire for TS support vs desire for ?

The sense in general as well as the trend from every other runtime is: Users are asking for better TypeScript experience a lot and almost no users asking for other type systems (like flow) or languages (like reason). Heck, someone opened an issue on this today

[benjamingr]

Also let me try pinging some TypeScript people

[mcollina]

When we say that users are asking for TS support, is there a sense for the ratio of desire for TS support vs desire for ?

Not at this point. I would be happy to evaluate other languages (like coffeescript), but I do not think any other language is as popular as TypeScript. Anyway, I think our implementation should allow for more extensions to be considered for inclusion.

[bnoordhuis]

More radical: bundle swc and just run the code. :-)

(Caveat emptor: swc is a transpiler, not a type checker. But on the flip side, it also handles JSX and TSX.)

[JakobJingleheimer]

More radical: bundle swc and just run the code. :-)

(Caveat emptor: swc is a transpiler, not a type checker. But on the flip side, it also handles JSX and TSX.)

That's not so crazy to me. It's sort of what I was suggesting above: #43818 (comment)

[manekinekko]

After seeing how other emerging runtimes provide native TypeScript transpilation by default, I am in favor of @mcollina's proposed idea. I'd like to see Node support running TypeScript code, ideally out of the box, with no additional flags or options.

[mcollina]

I'm in favor of @orta proposal. I have some questions:

Does this even work for node index.ts because it requires the files to be ESM meaning there must be a package.json - is that an acceptable trade-off?

Why? I don't see why this would be needed.

Performance oriented JS transpilers (sucrase/swc/esbuild/etc) would likely be faster than the "typescript" transpile APIs, but at the trade-off of perfect accuracy - this doesn't proclude them being used as loaders, but does prefer the built-in TypeScript. It feels like a push to say that node could have a list of their loaders are prefer them, but I think there'd be resistance in node.js to that given the dancing landscape of the JS ecosystem.

We should provide some configurability for this mechanism, something in package.json that allows us to map .ts to the swc loader.

[orta]

I assumed that because they were called ESM module loaders they would be ESM only - but if not, that makes a stronger case for this technique.

The configurability makes sense, as "typescript" would be default any other loader-ish package definition should be able to overwrite it 👍🏻

[mcollina]

I assumed that because they were called ESM module loaders they would be ESM only - but if not, that makes a stronger case for this technique.

I think there is a plan / possibility to extend them to CJS too.

[GeoffreyBooth]

Hey folks, I was in openjs summit’s node.js tooling group session yesterday where this came up and tried to come up with a new idea for how this could work.

Hi @orta, thanks for your ideas. A few thoughts:

• Per above, we have a few next steps already outlined: https://github.com/nodejs/node/blob/main/doc/contributing/maintaining-types-for-nodejs.md#high-level-approach—development-workflow. These are pretty minimal though (an error message that directs people to instructions) so there’s no reason we couldn’t start with these improvements and continue onto more dramatic enhancements later.

• Your idea sounds a lot like module: add support for nodeEntryPointConfig.loaders in package.json #43973, a PR for adding loaders configuration to package.json; note the last few comments on that PR. Those ideas (support .env file, or a Node configuration JSON file) could be ways to achieve what you’re after, without needing to add anything particular to TypeScript into the Node codebase. Something like an install script could automate adding the configuration.

• I wouldn’t want Node to treat a "typescript" dependency as a special case where its presence makes Node behave differently. Besides being potentially confusing (it seems to break the rules of what Node modules can do, if this one is treated differently than others) it also means that Node only has this enhanced TypeScript support when the primary "typescript" package is installed; but users might very well prefer to use esbuild or tsx or who knows what else for their TypeScript compilation and/or type-checking. However as long as we have npm install scripts, npm install typescript could easily prompt the user something like “add configuration to support TypeScript?” and then it could do something like add --loader typescript/register somewhere (package.json "scripts", a future .env or config JSON file). This would be a general solution that could work for anything that would want to customize Node via a loader (JSX, YAML, CoffeeScript, whatever). I know there are security concerns around npm install scripts, but npx commands could do the same work; there could be a TypeScript equivalent to npx create-react-app that would add the dependency and configure the options.

Loaders/Hooks/Customizations are still being figured out but having this as a launch goal could be a big ecosystem win, and could help shape their API?

Loaders are on the verge of being stable: https://github.com/nodejs/loaders#milestone-1-parity-with-commonjs. Adding support for some type of file-based configuration (as opposed to the current flag-based) is on the roadmap, but unnecessary for the Loaders API to leave experimental status because it shouldn’t be a breaking change to add a new way of defining configuration. There are no API changes needed to support TypeScript, as it works today via things like node --loader ts-node/esm. All that remains is making this easier, either through prompting the user with what to do or automating some or all of the steps to making it work.

Also, a solution along these lines, where we don’t special-case the typescript package, means that we don’t need any buy-in from the TypeScript team. We could solve this all on our own, and users could choose whichever TypeScript library they prefer.

[cspotcode]

There are no API changes needed to support TypeScript, as it works today via things like node --loader ts-node/esm.

This is a bit misleading since ts-node/esm installs several non-loader hooks into the main thread. Yes, it does work today, but future changes to node or to the loader API might break or hinder our ability to install these non-loader hooks. From a user's perspective, it's a loss of functionality when those hooks go away: TS that was working previously will start failing with runtime errors.

[GeoffreyBooth]

Yes, it does work today, but future changes to node or to the loader API might break or hinder our ability to install these non-loader hooks.

I’m unsure what this means. We’re not intending to change Node to break TypeScript support. On the contrary, making the Loaders API stable is meant to achieve the opposite: an API that can be relied upon, that won’t be removed or changed absent a Node semver-major change.

We have discussed adding similar customization APIs for things like source maps and the REPL and so on; I assume these are the “non-loader hooks” you refer to? The goal there would be to add APIs to allow tools like ts-node to customize those areas of Node in non-hacky ways that also provide more stability, as they would be officially supported Node APIs rather than unofficial monkey-patching or the like. But again, no one benefits from changes to Node causing breaks in expected behavior. The goal would always be to improve compatibility and reliability.

[cspotcode]

I’m unsure what this means.

We have to use undocumented monkey-patches. When node ships breaking changes to undocumented monkey-patchable functions -- perhaps accidentally -- there is sometimes resistance to fixing the breakage, since the API surface is undocumented. Stabilizing loaders is good, but it doesn't stabilize these undocumented hooks. From a maintainer's perspective, this risk is not fun.

One of those hooks is to give us proper module resolution: Module._resolveFilename It's not related to REPL nor sourcemaps.

I agree everyone's on-board to enable great TS support. I hope that until the loaders API fully supports the necessary hooks, the node team will treat these undocumented monkey-patches as an API surface that cannot be broken.

[GeoffreyBooth]

I hope that until the loaders API fully supports the necessary hooks, the node team will treat these undocumented monkey-patches as an API surface that cannot be broken.

The undocumented hooks you’re talking about are additional scope beyond the Loaders API, but something we intend to address with a new Customization API: nodejs/loaders#95. The goal is to support things like ts-node entirely through documented, stable APIs.

[knyga]

These days it is common to use services that do provisioning and management for you, AWS Lambda in particular. So they do not care that much about node.js runtime.
But developers do care. Developers are forced to transpile the code before deployment and have transpiling pipeline.

Secondly, I can't believe external transpilation could do better than native execution.

Thirdly, bun and deno already have this feature. Their runtimes want to stand out and get some audience based on unsolved needs. Isn't it a good sign that market expects TypeScript to be there?

[eduardofx]

I think we can have pre-defined values of tsconfig and the developer who wants to use other resources create inside the folder manually.

[vdeturckheim]

Ok, I started to experiment with implementation and here are my findings/opinions for tonight:

• the UX suggested by @mcollina works pretty well

 % ../node index.ts
The TypeScript compiler is not available.
Install it following documentation at https://nodejs.org/en/knowledge/getting-started/working-with-typescript/

then

 % ../node index.ts
1
number

if the @node/tsc-adapter module is available in the path - it can be local or global

• I don't have strong opinions on which transpiler to use. We can even allow multiple
• we might want to ship a very conservative default config (something that is expected to hopefully never change or break) and let users extend or override it
• we should probably prevent loading a .ts file from node_modules or at least allow it behind a flag, this can't go well because of potential configurations missmatches
• repl case will probably require a flag/entry in package.json if exist - that's probably ok - we probably might ignore TypeScript in the repl for the first iteration anyway?

% docker run -it --init denoland/deno:1.10.3 repl
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Deno 1.10.3
exit using ctrl+d or close()
> const x: number = 1
Uncaught SyntaxError: Missing initializer in const declaration

• we hit a similar limitation as Deno -> we need to have the relative modules have a .ts extension. I guess that since for 99.9% of the cases, semantically, TypeScriptTranspile(jscode) === jscode we could just ignore this problem?

I won't have coding time for a couple days, but should go back into writing stuff early next week

[GeoffreyBooth]

@vdeturckheim This looks good! A few thoughts:

• I don’t know what this proposed @node/tsc-adapter will contain, but I assume it’s a TypeScript loader like ts-node? If so it’s tied to the Loaders API which is still experimental, but Move ESM loaders off-thread #44710 is the last PR before it could become stable. Also this TypeScript support will surely be experimental for a while too, and longer than the Loaders API is, so that shouldn’t be an issue.
• Since the @node/tsc-adapter package would be installed and versioned separately from the Node runtime, it would have to support a range of Node versions. I think this is a problem already solved by ts-node and similar projects.
• Regarding package.json stuff, there was a prior discussion about including the loader settings (like what would be passed into --loader) in there, and I remember the consensus landing on including all of NODE_OPTIONS in there, or as much as is feasible. I think the way forward is to create something like a package.json "nodeOptions" field and add as many options as possible; it’s probably impractical to support all 100+ in a single PR, but we could at least come up with a design that can accommodate eventually supporting all of Node’s options and we can add them in batches.
• Requiring the .ts extensions isn’t so weird, they’re already required in ESM JavaScript generally. I know it’s unidiomatic for TypeScript, but support for explicit .ts extensions is coming in TypeScript 5.0 and we can provide an informative error on missing .ts extension like we already do for missing .js extension.

[vdeturckheim]

Thanks @GeoffreyBooth , these are very good points!
Yeah, @node/tsc-adapter is another name for typescript-node-core suggested in the first message of the thread and would be ts-node with node using it instead of having to plug into node. Basically, making the TS compiler live outside of node while allowing node xx.ts without any other argument. Right now, I made it work with cjs but for esm it will likely be a loader (we had experiments with @targos a few years ago on that too).

Regarding node vs. package version, this is a very good point.
I think we are going to a direction where the typescript support can work very well in most cases and where we provide DIY solutions for when it does not (like overriding the typescript adapter for instance).

For repl and .ts cases. I agree 💯 with you. In the end, these are best-effort and DX choices, we can take the decisions and document them. Also, this is what experimental support is for!

I missed last summer's summit (and this thread until recently fwiw), so I hope I am still going in the direction people agreed upon 👼

[GeoffreyBooth]

I missed last summer's summit (and this thread until recently fwiw), so I hope I am still going in the direction people agreed upon 👼

Yes, I think nothing has changed from #43818 (comment) and what your recent experiments seem to assume. About the only “news” is that we’ve made so much progress with the Loaders API that I can see it applying to CommonJS too in the not-too-distant future; so I would prioritize getting TypeScript support via Loaders working first or at least ship that at the same time as CommonJS solution. @cspotcode and others have also had discussions about hooks in addition to those provided in the Loaders API, to do things like customize the REPL, for a more seamless DX beyond loading modules. Hopefully we can use the interest in better TypeScript support to build out some of those APIs.

It sounds like package.json options aren’t needed for the core/MVP use case, so maybe we should get the initial to-do list done and then we can loop back to that and give it the attention it deserves.

[vdeturckheim]

This sounds good, thanks for the highlights! Seems good to me! I'll resume work with that in mind!

Regarding the point in not treating "typescript" differently (in #43818 (comment)), I agree, that's one of the advantage of having something like @node/tsc-adapter which we control and would only serve the goal of calling "typescript" from node under the hood on .ts files. But nothing prevents to also have @node/swc-adapter or @node/esbuild-adapter tooto let the end user chose the transpiler in a list of provided alternatives (these packages are proxies to the other packages which also abstract breaking changes such packages could have). The end user will anyway be able to use a custom loader if they want to use something else in the end.

But let's focus on the MVP for now and add too much complexity later 👼

[tony-go]

But nothing prevents to also have @node/swc-adapter or @node/esbuild-adapter tooto let the end user chose the transpiler in a list of provided alternatives (these packages are proxies to the other packages which also abstract breaking changes such packages could have). The end user will anyway be able to use a custom loader if they want to use something else in the end.

+1 on this ^^

[GeoffreyBooth]

But nothing prevents to also have @node/swc-adapter or @node/esbuild-adapter tooto let the end user chose the transpiler in a list of provided alternatives

I’m not sure Node needs to publish a transpiler at all. Whatever docs page our error message links to could include a list of ones known to work, and encourage the user to choose one to install. I would think we should at least start with that approach, and if we find a need for publishing something official we could do that after MVP.

[hinell]

There is a related proposal to add type syntax to JS, checkout:

• https://github.com/tc39/proposal-type-annotations

[shinebayar-g]

What is the latest update on this? With both Deno and Bun improving their compatibility with Node.js and providing native support for TypeScript, there seem to be numerous examples and lessons to be learned from their advancements.

Since it hasn't been mentioned in this thread, I would like to inquire whether this proposal could enable the distribution of npm packages exclusively written in TypeScript. Such a capability would significantly enhance the user experience when working with Node.js and TypeScript, eliminating the need to juggle between .d.ts and .js files.

[GeoffreyBooth]

What is the latest update on this?

We’re exploring this space via #49704 (cc @JakobJingleheimer). One idea we’ve been considering lately is that Node could possibly provide some kind of blessed or automated way to install plugins, one of which could be TypeScript. Part of the problem is that TypeScript itself doesn’t provide Node integration, and there are many third-party plugins for that purpose, and we would want to avoid picking a winner. There’s also no one correct way to set up TypeScript; it depends on your build tool and its configuration, your app and its framework and those expectations, and so on.

You might want to open an issue with the TypeScript team, if there isn’t one there already. They’re the ones who have the power to add official support for runtime transpilation and/or type-checking, and/or provide a blessed/recommended tsconfig.json for Node projects. TypeScript itself is somewhat coupled to Node, in that the extensionless resolution mimics Node’s CommonJS resolution and the conversion of ES modules to CommonJS modules was created to address former limitations of Node; the TypeScript team is perhaps the best source for how they advise integrating their tool with Node.