[JavaScript] Use prototype for comments

[Thom1729]

(This PR formerly known as #1003)

Background

The default JavaScript syntax definition is descended from an old .tmLanguage. Since its transition to a .sublime-syntax, it has largely modernized, but signs remain of its history.

One such sign is that the syntax definition does not use the new prototype feature for comments. Instead, the comments context is included manually. There are many dozens of contexts in the definition and somewhere between many and most of them should have manually included comments. An unknown number of those did not, resulting in at least one bug (#885, first test case).

In addition, the tests did not include any negative tests to ensure that comments were not included where they should not have been. No related bugs are known.

This commit

I have moved the comments include to the prototype and removed it from all other contexts. I added meta_include_prototype: false as appropriate to remove it from the few contexts that should not include it. Finally, I have added negative tests to verify that comments should not appear in those contexts and a test for the known bug that was fixed in the process.

Discussion

The known bug could have been fixed by adding another manual include. So why make a larger change?

The primary reason is simplicity. Comments are a pervasive feature that applies to nearly every part of the language. Each - include: comments is a separate implementation of this feature. While not every context would need that include, it is necessary for every context to consider whether it is needed. In at least one known case, the include was mistakenly omitted (and there are probably more such cases).

Conversely, the parts of the language that do not need to support comments are few: strings, regexps, and comments themselves. These parts all stand out as special cases with their own unique microsyntaxes. Additionally, this enumeration is likely to be relatively stable as the language evolves. Every piece of new syntax may require a new context and explicit consideration of comments, whereas new types of strings don't come along very often.

Prototypes also lessen the testing requirements. In this commit, I added a test for each meta_include_prototype: false that covers all comment types. Without prototypes, there really ought to be such a test for every single context that does allow comments. Many, many tests would be required to catch bugs such as the example from #885. Such tests do not yet exist.

In short, I believe that it makes the most sense to mark each exception rather than each non-exception. Doing so will result in less code and require fewer tests.

Remarks

• There seem to be three separate copies of single- and double-quoted strings. These could perhaps be consolidated.

• This change is one of the steps on the program of improvement I outlined in [JavaScript] Division incorrectly highlighted as regex. A general fix is known, but nontrivial and could use feedback. #885. I consider this PR to be an important part of the foundation for this program.

[wbond]

Thank you – I appreciate the explanation of what and why.