Feature Request: Source maps

[SLaks]

It would be great to click in the preview pane in a Markdown editor & select the piece of source Markdown that rendered to that text.

Can you add an option to generate source maps so that editors can support this?

This is also necessary for synchronized scrolling.

/cc @am11 @madskristensen

[am11]

I second that! 👍

There are also couple of other advantages mentioned by @kizu here: markedjs/marked#382. Whichever compiler would support source-map first, we should probably go with that one. Currently, no markdown compiler supports source-map feature.

If this interests the dev(s), please consider capturing all the nitty-gritty mappings to avoid this kind of issue: sass/libsass#324. A good criterion would be to make sure each token in the generated output code has a mapping, which corresponds to the original source.

[jgm]

The parsers already have source location information in the abstract
syntax tree, so it's a fairly trivial change to put this in the HTML
output. (I've done this as an experiment.) This may not be the same
as what you're calling a source map, but the information is all there
and I'm sure it could be extracted in whatever way you like.

[am11]

It's a standard mapping standard called V3 source maps: https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit?pli=1#!, which browsers like Chrome, Firefox and IE support. There are couple of variants, the more significant ones are: inline source-maps and encoded source-maps VLQ Base64 (vlq=variable length quantity).

If the AST has this context, then its just a matter of adding those two variant for the starters. 😄 👍

[mb21]

@am11 what you reference are a different kind of source map, used when compiling from some other language (e.g. coffee script, dart, clojurescript etc) to javascript. Then the source maps in browsers' debug tools are pointers from the javascript back to the code in the original programming language.

I don't think this has anything to do with what OP meant: i.e. sourcemaps from the generated HTML back to the original markdown.

[SLaks]

@mb21 No; mapping from generated HTML to original Markdown is no different from mapping from generated JS to original CoffeeScript.

[mb21]

@SLaks I see, so the advantage of using source maps, as opposed to a custom format, would be that even though you aren't using the browser's dev tools, you could take advantage of tools like mozilla's source-map?

[SLaks]

Yes, and, in particular, our existing editor code.

[mb21]

I guess that makes some sense. However, if someone were to create a markdown editor from scratch, wouldn't it be more advisable to embed the 'source maps' in the HTML with attributes like data-sm-line=2 data-sm-column=20, referencing line 2, 20th character in the markdown file? If I understand correctly, a lot of the complexity in source maps arises from the fact that you cannot embed additional information in a plain-text source code and therefore need an additional file. However, when generating HTML you can link each element back by attaching an attribute. Or am I still missing something?

[kizu]

@mb21 You can inline source maps as comments in most implementations, so it would be possible to inline a map to HTML in the same way.

And you can't mirror source maps using HTML attributes, as different parts for one HTML tag could come from different places and/or even different files. Not to mention that that would be reinventing a wheel, while there is already a standard way of doing it: by using source maps.

[mb21]

I agree that it's generally bad to reinvent the wheel. But isn't the usecase of markdown->html different enough to consider a better solution? For example, I'm not aware of any markdown editor that lets you combine multiple markdown files to generate a single html file (without resorting to external utilities like the shell). Similarly, can you post a markdown snippet that generates HTML where one tag comes from different places in the markdown file? It seems to be in the nature of markdown that there's quite a one-to-one mapping there.

[kizu]

Markdown has, at least, reference links:

Hello, [world][1]!

[1]:
    http://example.com/
    (example site)

This would generate this HTML:

<p>Hello, <a href="http://example.com/" title="example site">world</a>!</p>

You can see that href and title attributes come from another place, so it wouldn't be enough to have a single line/column anchor.

And while currently there are just few examples like this, there could be extensions that would allow you to use different files and do other things.

And another point: source maps are made in a way they could be changed to always represent the correct source, so if all the tools in a chain support source maps, you still would be able to get to the source from the result. For CSS, for example, if you're generating CSS usin Sass of Stylus, then you can use Autoprefixer on the result, and then use CSSWring on the result from the autoprefixer — and as all the tools in this chain support source maps, you still could see where different things come from, even while the resulting CSS was modified.

Source maps on their own wouldn't make much sense, but with other tools — they would. The HTML generated from markdown could be then used in some HTML preprocessor like HAML or Jade, then it could be minified by some external tool, there could be some other changes etc. If all tools would accept source maps, then it would be possible to trace where each symbol in the resulting HTML comes from.

[am11]

Besides, source-map v3 already covers all that. Even if it is 1:1, sourceMap is the right way to tackle it.

There are couple of things you can do:

The HTML can contain:

...
</body></html>
<!--# sourceMappingURL= path/to/generated/sourcefile.map -->

And the sourcefile.map can point to .md file.

The mappings array in the sourcemap would contain 1:1 mapping for every token (tag, attribute, attribute-value, innerText) in the generated output, pointing to its counterpart in source code.

The map file can also contain the source code (.md files in this case): sourcesContent array. This is useful when source files are not hosted.

LESS compiler provides even more options. You can use --source-map-map-inline switch, which would encode the sourcemap file and serve it as B64 URL:

body {
  color: red;
}
/*# sourceMappingURL=data:application/json;base64,<B64-encoded-JSON-code> */

These are all the source-map related options provided by less npm compiler:

 --source-map[=FILENAME]  Outputs a v3 sourcemap to the filename (or output filename.map)
 --source-map-rootpath=X  adds this path onto the sourcemap filename and less file paths
 --source-map-basepath=X  Sets sourcemap base path, defaults to current working directory.
 --source-map-less-inline puts the less files into the map instead of referencing them
 --source-map-map-inline  puts the map (and any less files) into the output css file
 --source-map-url=URL     the complete url and filename put in the less file

[mb21]

@kizu Yeah, if you're looking at the scenario of inspecting the generated HTML source code, then your points make a lot of sense. However, I was thinking of a markdown editor with md on the left and rendered HTML on the right: when a user clicks on an image on the right, the editor scrolls and moves the cursor to [1]: in your example. There is no way for the user to click on the title or href attributes separately (minifying/converting the HTML doesn't change that).

P.S. agree that thinking ahead and supporting multiple files is a good idea, also since it doesn't add a lot of complexity.

[zdne]

Jumping into the discussion – as an author of a language based on Markdown and the subsequent tooling – Source maps were one of the biggest issues for us in the beginning. They play a crucial role in the interaction with our users.

The interaction happens through the (C/C++) parser's subsequent tooling (various languages) – usually to display semantic warnings in the markdown editor or CLI tools.

It took me quite a while to hack the support for source maps into the original sundown parser. Currently every Markdown block entity has the source map information associated

I would like to point out that in Markdown a source map does not have to a continuous block. Consider following:

Lorem Ipsum
Dolo Sit Amet

Clearly the paragraph has a non-continous source map using the <character index>:<length> format

The whole quite block would have 0:30 source map while the paragraph would be composed of 2:12 and 16:14

As for exposing the source map to subsequent tools – on demand we are exporting both the language AST and separate source map tree (1:1 with the language AST).

Sorry, if I am sharing obvious, just wanted to point out some pitfalls I have been through.

[colinodell]

+1

[jgm]

A lot of useful discussion here, but this doesn't belong in the repository for the spec.