The AT HTML parser does not handle comments as the framework does

[ymeine]

Aria Templates handle comments as a layer on top of the source code.

In practice it processes the comments (and some other special characters - escaping in general) before actually parsing the source: it strips them off. It doesn't take into account any context of where the comments are, it doesn't want to make assumptions about the language used underneath.

The problem with this is that comments can occur anywhere, even inside a string for instance (most common pitfall).

Requirement: the parser for the editor should be able to handle the comments the same way, and take them properly into account.

Solutions

Handle it in the grammar - rejected

Handling it in the grammar brings two issues:

• it's almost impossible because the comment should be put as an alternative for every other single rule, to be sure to catch them everywhere
• it doesn't make much sense, since it's a level above the language, we should keep processing it the same way

Use the concept of multi-language

This idea is inspired from the actual implementation in the framework, which first handle comments only, as if t were a language on its own, and then handles parts between comments with its parser.

Indeed, we can consider comments as being part of a very simple language defining only how to write comments. What's inside comments delimiters would be free text, and what's outside would be _unparsed_ text.

The parser of this language would then use the AT HTML parser to parse this _unparsed_ content.

This is the exact same principle that would be applied if the AT & HTML parsers were decoupled, and the AT one would use the HTML one to actually parse the HTML.

The main problem coming with that are the same:

• is the input for the nested parser correct after removing the parts corresponding to the current parser? (i.e. after removing comments, is it valid AT-HTML?)
• how to handle positions? Because by removing parts, we fake positions, but in the case of the editor, positions are not optional.

[ymeine]

Just to be a bit more accurate, comments stripping done by the framework does consider some contexts, but really a few: for now only double slash in URLs.

But this changed with ariatemplates/ariatemplates#772