Accept Input Source Maps

[LucasSloan]

The typescript compiler produces sources maps mapping from the javascript it produces to the typescript sources it compiled. This is great, but doesn't go far enough if there are tools which generate the sources that are consumed by the typescript compiler.

For example, Tsickle preprocesses typescript sources before passing them to the typescript compiler. As things stand we've had to implement a hack to get our source maps to work. We use a compiler host which performs the tsickle transforms as the typescript compiler reads files, saves the resulting source maps and then compose them with the source maps produced by the typescript compiler as they're emitted.

Instead the typescript compiler could accept source maps as input alongside sources, and compose the input source maps with the source maps produced by the compiler.

The ideal api would be that the typescript compiler would look for either a. an inline source map as a comment in the source file or b. a file in the same directory with the name {source file name}.ts.map, which shouldn't require any crazy file resolution.

[mhegazy]

For Tsickle, i would say the better way is to implement it as a transformation on the compiler. this saves you the double pass in the compiler. #13940 exposes the APIs needed to achieve this.

[LucasSloan]

Tsickle may not be the best example. The actual motivating reason for this issue is the Angular compiler is going to produce source maps in typescript it compiles from templates (html).

[nippur72]

Another example is the html to tsx a templating tool that I use (yet to be released). Without sourcemaps debugging is rather difficult.

[Daiz]

This is also something I would be interested in - I'm working on a TypeScript project that involves writing a small DSL which I would like to transpile to TypeScript code. With input source maps I could presumably just write a source map like you'd do for any other transpiler, and then TSC could use it as input so that the final JS could be mapped nicely back to the original DSL code.

While I could always just make the DSL compile directly to JS too, I would really like to do it with TS instead so that the code can be type checked for possible scripting errors (as the DSL interacts with TS modules).

[ygoe]

I was looking for some tool that can bundle any kind of multiple source files into a single file, transpile new JavaScript to something suitable for today's browsers and then minify all that. uglify-es can't bundle or transpile. rollup can't transpile or minify. babel can't do source maps. tsc can't bundle or do source maps either. I'm about to give up on JavaScript. Or I might revert to some stupid text-replace bundling (which is really just some full-file include) and pass that to uglify, but I'd need to create a source map for that myself then. Not production-ready.

[zaoqi]

@ygoe There is a tool for concat multiple source files
https://www.npmjs.com/package/tool-cat-with-sourcemap

#!/usr/bin/env node

const {SourceNode, SourceMapConsumer, SourceMapGenerator} = require('source-map')
const fs=require('fs')
const assert=require('assert').strict
const args=require('yargs')
      .usage('Usage: $0 -s [output/sourcemap] -o [output] <file> ...')
      .help('h').alias('h', 'help')
      .example('$0 -s sourcemap.json -o c.ts a.ts b.ts','concat a.ts and b.ts, write c.ts and sourcemap.json')
      .demandOption(['s','o'])
      .argv
const files=args._
assert(files.length>0)

const sn = new SourceNode(
    1,
    1,
    "global",
    files.map(file=>new SourceNode(1,1,file,[fs.readFileSync(file,'utf-8')])))

const x=sn.toStringWithSourceMap({file: ''})
fs.writeFileSync(args.s, x.map.toString())
fs.writeFileSync(args.o, x.code)

[nmain]

As mentioned in #34608, it would be useful if tsc, when accepting input sourcemaps, not only generated output sourcemaps based on them, but provided error reporting that used the sourcemaps as well.

[andersekdahl]

Here's another use case:
We use tsc to compile our TS code to js, which we then deploy to our production servers. But before actually sending it to the user we run a second pass (on the production server, not the build server) on the compiled and minified code to inject translations into the files.

Since we're already heavy users of TS transforms it would be handy to run this process with TS. But for that to be viable we'd need TS to accept the source maps generated during build as input.

[foreverflying]

Hi guys:
2 weeks ago I was facing the same problem and trying to find the solutions. The search brought me to this topic, and then I realized after so many years the problem is still not well solved.

So I decided to solve it myself. Now I finished this minify-ts.

github link

It can minify the ts file, and merge the source map files generated in several steps.
The minifying is slow because it relies on the TypeScript Language Service find-reference feature.
But the merging implementation is quite efficient, it can batch process all the source-map files in a folder, climb up through every source-map chain to the top to collect information, and use the merged result to overwrite the tail one.

Welcome to have a try and report bugs. Hope this solves your problem.

[rictic]

Running into a use case that would be well served by this today when converting YouTube's HTML template files to TypeScript at build time

[brad4d]

I've discovered that input source maps were also requested in #26843

I wonder if there are more...

The tsickle use case no longer matters to us at Google, but there are others and more to come.
We would like to generate TypeScript files, but we need to have source maps that refer back to the original source from which they were generated.

Surely we are not the only people interested in this. TypeScript is mature enough now that surely lots of people would like to automatically generate it.

We would likely be able to provide some engineering effort.
Would the TypeScript team be willing to collaborate on a design and implementation?

[azizghuloum]

Me too.

I'm writing a term-rewrite (hygienic macros) system for typescript. During rewriting/expanding a file, the plan is to keep track of the source of each node in the AST whether it originated from the same file or came from another file. The end AST (representing typescript code with no macros) can have a mix of sources. My initial thought was to produce the .ts file with sourcemaps and let tsc/tsserver/whatever deal with the .ts files either for type checking or bundling or whatever they wanna do with it.

Without input sourcemap supprt, I will have to:

1. integrate very deeply with the typescript compiler for the purpose of producing meaningful error messages and sourcemap files (both of which I have to map back to the user's initial code). This is wasteful and fragile.
2. produce 4 files .js/.js.map/.d.ts/.d.ts.map instead of just a .ts file with sourcemap. This means that I went from being a term-rewrite for typescript to being a full compiler. This is really out of scope of rewrite-ts.

So no, can't do that. For now, I will keep my project's scope and produce .ts files only, hoping that this issue is resolved in time.

Lots of projects today compile TO typescript and more will come in the future. Supporting input sourcemaps in the typescript compiler will help all of these projects.

Thanks for all the good work.

---

• https://github.com/azizghuloum/rewrite-ts-visualized/ (repo)
• https://azizghuloum.github.io/rewrite-ts-visualized/ (playground)

[jakebailey]

FWIW if you're already generating .ts and .ts.map files, you can combine TypeScript's .js.map output files and your provided .ts.map files with https://www.npmjs.com/package/@ampproject/remapping; this package is exactly what Babel uses to implement its own input source map functionality.

See also:

• https://github.com/babel/babel/blob/e66517c1765cf6fb2c39ea0525b1a5d2141ab421/packages/babel-core/src/transformation/file/generate.ts#L65
• https://github.com/babel/babel/blob/e66517c1765cf6fb2c39ea0525b1a5d2141ab421/packages/babel-core/src/transformation/file/merge-map.ts#L2

[azizghuloum]

@jakebailey right, but that's assuming the end destination is a js file on disk and that I want to take care of the entire pipeline. I honestly have no idea what people would do with the ts file. I want to generate it and get out of the way.

If I'm generating .ts file with embedded sourcemaps, it might be ready for use by most tools today (say, deno, bun, vite, etc.) that would strip out the types and bundle/stream/run the code and probably respect the embedded sourcemaps. In the future, browsers would also load the ts files with their sourcemaps and do the right thing. The microsoft typescript compiler is not the only consumer of ts files today.

Also, if I'm generating the .ts file with embedded sourcemaps, and the file goes to tsc or tsserver, any diagnostics produced should be in terms of the original source locations, not the generated source code, just like how runtime stack traces go back to the original source, not the generated ones.

[maskmaster]

What many tools need to do is being able to map a location (e.g. from an error message) to the original document position.
This can be achieved outside of TypeScript by mapping the location via the sourcemap to the original location.

However, this is something many different tools need to manage by themselves, and it is kind of a hassle. The complicating matter is that it could be hard for each tool-developer to identify all places where TypeScript are exposing source code locations, and to remember to map those locations. (Or even worse, if the location would end up in text form in some message, the consumer would have to parse the message format)

I would classify this as a convenience feature, but I can imagine that most tools generating typescript code would in some way need this functionality (DSLs, TS-generating tools, etc.).

I need it for two tools I am maintaining.

[brad4d]

I believe one of the reasons that closure-compiler ended up supporting input source maps was the need for it to produce good error messages itself. It's not helpful for closure-compiler to generate an error message that includes a reference to and snippet of a generated file that the user didn't actually write and which may be very opaque to them. It's much more helpful for the error message to point to the code the user wrote.