[Work in Progress] Initial work on native sublime syntax support #109

skyronic · 2017-11-14T16:37:37Z

What works:

Basic template/script/style tags
Limited subset of extra languages supported (sass, scss, pug, coffee)

Still to do:

Investigate stability and compatibility with older versions, etc
Support all languages

To test this out:

Disable existing Vue
Download Vue Component.sublime-syntax file from here
Put that file somewhere in the sublime text user folder

If you encounter issues with modern JS features, install the Babel Syntax highlight package and disable the built-in JavaScript.

yyx990803 · 2017-11-14T21:14:29Z

Thanks for working on this @skyronic !

I tried it out locally, but running into a few issues:

expressions inside mustaches are not highlighted as JavaScript
<script> and <style> hightlights are not being highlighted properly

Thom1729 · 2017-11-14T22:35:40Z

This syntax handles attributes like the core HTML syntax, but the core HTML syntax has some problems in this regard. I submitted a PR today addressing some of these problems; you might want to take a look at it.

The syntax doesn't handle attributes in <template>, <script>, and <style> tags, except for lang. Are other attributes (such as id) not allowed in Vue components?

The way that Vue directives are injected into the HTML syntax seems problematic:

<template>
    :foo="
</template>

Surely that's not supposed to happen. I can see how you're trying to avoid getting into the weeds of the HTML syntax with a simple include, but I don't think it's going to work. I wish I knew of another easy answer that didn't involve copy-pasting the HTML syntax and starting from there, but all I have to recommend is my YAML Macros package.

skyronic · 2017-11-15T03:49:31Z

@yyx990803 I've tested this out with the latest dev build of sublime. Will test it out with stable version and update the PR.

@Thom1729 There are a small set of attributes for <template> , <style> etc but not all HTMl attributes. Those aren't html tags (even though style and script are in HTML) that's why I declared the main scope as text.vue rather than html like the Jinja and ERB syntaxes.

One downside is that attributes aren't colored properly so I'm catching the most used attributes and attaching scopes to them as a bit of a hack. Will keep updating this. Will look into the option of embedding html syntax as well.

skyronic · 2017-11-15T06:18:02Z

@yyx990803 looks like the issue is I was using some features of sublime-syntax which was only available in dev (embed/escape).

While this is arguably a better and cleaner way to do it than the older approach, it looks like it's simply not going to be feasible until everyone updates to the latest version.

My current idea is to try an approach which works for older and newer versions, and we can consider upgrading to the correct approach later.

skyronic · 2017-11-15T09:52:52Z

@yyx990803 I've update it to use the backward-compatible approach which should work with the current stable release.

Still many language support yet to be done, and there might be bugs. Please do try it out when you've got some time. It should be ready for release soon.

There's a new feature embed and here's a pretty significant benefit it offers. For example, currently, the highlighting breaks if there's an error in the JS (but works fine if there's no syntax errors):

But if we move to the embed system which is available in the dev build, we shouldn't have this problem. However, since the current version of vue-syntax-highlight also has the same issue, I think it should be fine. I'll still explore if there's a feasible workaround to improve this behaviour.

yyx990803 · 2017-11-15T14:36:43Z

Thanks for the update @skyronic - I didn't realize embed/escape is only available in the dev build.

Coincidentally, the current syntax still works fine in the stable release and only breaks in the dev build. Maybe we should just go with the new mechanism in this PR as we work on it, and release it as a separate package so that users of both stable and dev can get a working syntax?

Thom1729 · 2017-11-15T14:54:07Z

You can modify the Package Control channel record to declare that certain releases are compatible with 3153 and below and others are compatible with 3154 and above. This way, Package Control won't upgrade users on an older Sublime build to a version that depends on embed/escape. Or you could use with_prototype and punt on embed/escape for now. The major downside to this is that it bloats the compiled syntax, particularly in a language like Vue that includes many other syntaxes.

In either case, you should make sure you're developing on the dev build, as the internal JavaScript implementation has changed substantially; #107 does not occur at all on older builds.

skyronic · 2017-11-15T14:58:16Z

(deleted an older comment since @Thom1729's message clarifies my question)

Thom: a new question. If a user on an old version of sublime, who doesn't have the extension installed tries to install it, does it fetch the appropriate version?

Thom1729 · 2017-11-15T19:43:01Z

Yes; that is my understanding. So if today you released a new version for 3154+, and a user on the stable channel installed this package, they would get the old version, whereas a user on the dev channel would get the new version.

z3nz · 2017-11-16T03:40:41Z

Hello there. I know this is still in progress, but I thought I would share a couple of the bugs that I've come across while using it. This is for the most recent commit as of posting this.

In the screen shot there are 3 bugs:

Syntax highlighting persists in commented code
Template elements inside of a template element breaks the highlighting after the first closing tag
Highlighting breaks before the closing script tag

I've noticed that the third can be fixed by adding a semi-colon after the closing mustache of the export, but not including a semi-colon is also correct (especially since I'm using the standard coding style).

Thom1729 · 2017-11-16T04:49:31Z

Semi-relevant question.

not including a semi-colon is also correct (especially since I'm using the standard coding style).

What coding style is this? Is there a standard coding style that recommends omitting semicolons?

skyronic · 2017-11-16T06:25:00Z

What coding style is this? Is there a standard coding style that recommends omitting semicolons?

A lot of them really, and most teams I've worked with over the last few years omit semicolons. https://standardjs.com/ is one of them. The problem is export default has this problem but export {} doesn't.

skyronic · 2017-11-16T13:33:58Z

@yyx990803 so here's my investigation on how we can ship a new version. In the package control source code there's some instructions

		// If your package is only compatible with specific builds of
		// Sublime Text, this will cause the package to be hidden from
		// users with incompatible versions.
		//
		// The "tags" key can be true for all valid semantic version tags, or
		// can be a prefix string. Only tags in the form
		// {prefix}{semantic_version} will be selected. In the example below,
		// the entry with "sublime_text": "<3000" will match tags like:
		//
		// "st2-1.0.0"
		// "st2-1.1.0"
		//
		// The release with "sublime_text": ">=3000" will match tags like:
		//
		// "st3-1.0.0"
		// "st3-1.1.0"
		{
			"details": "https://github.com/wbond/sublime_alignment",
			"releases": [
				{
					"sublime_text": "<3000",
					"tags": "st2-"
				},
				{
					"sublime_text": ">=3000",
					"tags": "st3-"
				}
			]
		},

I looked through the package control repository and found a suitable candidate which followed this pattern:

		{
			"details": "https://github.com/SublimeText/NSIS",
			"labels": ["language syntax", "nsis", "build system"],
			"releases": [
				{
					"sublime_text": "<3103",
					"tags": "st2-"
				},
				{
					"sublime_text": ">=3103",
					"tags": "st3-"
				}
			]
		},

I located an old build of Sublime text (3102) and installed package control. I then tried to install that extension and it found the correct version and installed the old format.

So here's what I propose:

We make a new branch for the "new" format.
Add a rule to only install that tag for the new format, and use the embed/escape format in it.
Make a release before the dev build of ST becomes stable and users get notifications (@Thom1729 are you familiar when the new stable will release)

Does this sound ok?

yyx990803 · 2017-11-16T13:51:00Z

@skyronic sounds like a plan!

Thom1729 · 2017-11-16T14:43:03Z

Unfortunately, I have no special insight as to when a new stable build will be released. It could be today or it could be several weeks. I would guess that we are likely to have several more dev builds first, as there are some corner use cases for escape/embed that wbond may be addressing.

The problem is export default has this problem but export {} doesn't.

The export {} form requires a semicolon as well. The difference is that there are no possible valid tokens after the closing brace other than the semicolon, so the automatic semicolon insertion algorithm will insert a semicolon in more cases.

This reverts commit 3beb4ba.

skyronic · 2017-11-19T10:25:58Z

@z3nz thanks for the failing cases, they are fixed now!

Thom1729 · 2017-11-19T17:17:36Z

This looks much improved. Some comments I have at this stage:

Most syntaxes used hyphens in context names instead of underscores, e.g. template-langs.
It looks like there are two implementations of HTML comments, one of which is more complete than the other. These could probably be factored out.
Attributes other than lang aren't handled. It is impossible to parse an open tag in one expression while handling attributes; you might have to rework your approach here.
Line breaks in various valid places would disrupt the highlighting.
The implementation of Vue directive attributes looks like it's based on the HTML attribute implementation. However, that implementation has a number of problems. Take a look at this PR for a better way.
In addition, the embedded HTML looks like it looks for Vue directives even outside open tags.
Also, the scope for the attribute name includes event, which is not appropriate in general.
The {{ and }} punctuation are not scoped. The appropriate scope is probably punctuation.definition.template-expression.[begin|end].vue.

skyronic · 2017-11-20T13:01:12Z

@Thom1729:

Thanks for your notes. I've incorporated many changes with the suggestions. Just wanted to confirm a few things:

In addition, the embedded HTML looks like it looks for Vue directives even outside open tags.

Yeah. I noticed this is an issue. And would appreciate your insight in helping me figure out a solution. Right now I'm doing it like this:

    - match: (?:^|\s+)(v-\S+|:\S+|@\S+)\b\s*(=)\s*(")\s

However, if I do it something like this to only match a html tag

    - match: (<).* ... snip ...

I will end up having to do attribute parsing and re-implementing the whole of tag-stuff from the HTML part. I'm relatively new to this so if you can suggest the correct approach to go about "extending" the attributes without reimplementing and matching the HTML behaviour that'll really help me.

As for the <style> and <script> tags, they aren't full HTML attributes and again, this is the same issue as last time. I don't want to re-implement the attribute behaviour.

- changed underscores in names to hyphens - added support for single quote in attribute strings

skyronic · 2017-11-20T13:39:16Z

Wrote a post in the sublime forum asking for inputs regarding this issue.

Thom1729 · 2017-11-20T14:22:02Z

It seems like what we need to do, in a nutshell, is extend the HTML syntax with additional functionality. The base syntax highlighting system doesn't make it easy to do this; generally, you'd have to copy the entire HTML syntax and then manually modify it. The best (only) alternative I am aware of is my YAML Macros package. I'll post a tailored example later.

skyronic · 2017-11-20T14:55:18Z

@Thom1729 my concern is having to manually update and track changes to the "official" sublime text html package. Will wait for your inputs.

@yyx990803 these are the issues right now:

Some vue related attributes get highlighted outside of a HTML attribute:

Attributes in style/script tags etc don't get highlighted (apart from lang)

Newlines break the syntax (not sure why this is happening but it should be an easy fix):

Thom1729 · 2017-11-20T20:08:30Z

Here's an example:

%YAML 1.2
%TAG ! tag:yaml-macros:YAMLMacros.lib.extend:
---
!extend
_base: HTML.sublime-syntax
name: Vue
scope: text.html.vue
hidden: false

contexts: !merge
  main: !prepend
    - include: vue-tags

  tag-attributes: !prepend
    - include: vue-directive

  vue-tags:
    - match: '(?i)(<)(template)\b'
      captures:
        0: meta.tag.template.begin.html
        1: punctuation.definition.tag.begin.html
        2: entity.name.tag.template.html
      push:
        - match: (?i)(</)(template)(>)
          captures:
            0: meta.tag.template.end.html
            1: punctuation.definition.tag.begin.html
            2: entity.name.tag.template.html
            3: punctuation.definition.tag.end.html
          pop: true
        - match: '>'
          scope: meta.tag.template.begin.html punctuation.definition.tag.end.html
          push: mustache-template
        - match: ''
          push:
            - meta_scope: meta.tag.template.begin.html
            - match: '(?=>)'
              pop: true
            - include: tag-attributes

  mustache-template:
    - match: (?=</template)
      pop: true
    - match: '{{'
      scope: punctuation.definition.template.begin.html
      embed: scope:source.js
      escape: '}}'
      escape_captures:
        0: punctuation.definition.template.end.html
    - include: main

  vue-directive:
    - match: \b(v-[\w-]+)\b
      scope: entity.other.attribute-name.html
      push: js-attribute-value
    - match: \B(:|@)([\w-]+)\b
      captures:
        1: punctuation.definition.attribute.html
        2: entity.other.attribute-name.html
      push: js-attribute-value

  js-attribute-value:
    - match: '='
      scope: punctuation.separator.key-value.html
      set: js-string
    - include: else-pop

  js-string:
    - match: '"'
      scope: punctuation.definition.string.begin.html
      embed: scope:source.js
      escape: '"'
      escape_captures:
        0: punctuation.definition.string.end.html
    - match: "'"
      scope: punctuation.definition.string.begin.html
      embed: scope:source.js
      escape: "'"
      escape_captures:
        0: punctuation.definition.string.end.html
    - include: else-pop

This doesn't implement everything, but it does solve the core problem of coherently extending the HTML syntax with new behavior.

This example is based on this PR, which fixes some of the HTML syntax's whitespace issues. In order to build, you need a local copy of that syntax. When the core HTML syntax is updated, you would pull a copy of that upstream syntax and rebuild the Vue syntax. This is admittedly inconvenient, but it's certainly less inconvenient than copying the HTML syntax, extending it in-place to handle Vue, and integrating future improvements in the upstream HTML syntax manually in the patched syntax. I am exploring ways to improve the experience of using YAML Macros to extend built-in syntaxes and would welcome input along these lines.

I recommend YAML Macros fairly frequently because I frequently run into situations where the only way to extend an existing syntax in a certain way is by copying it and patching the copy. YAML Macros isn't an alternative to this approach, but rather a framework that avoids some of the pain of doing it manually.

skyronic · 2017-11-28T13:29:03Z

@Thom1729 Unfortunately I was not able to build the example you listed with Yaml Macros package. I get this error:

  File "/Users/anirudh/Library/Application Support/Sublime Text 3/Packages/ruamel-yaml/st3/ruamel/yaml/representer.py", line 111, in represent_data
    node = self.yaml_representers[None](self, data)
  File "/Users/anirudh/Library/Application Support/Sublime Text 3/Packages/ruamel-yaml/st3/ruamel/yaml/representer.py", line 375, in represent_undefined
    raise RepresenterError("cannot represent an object: %s" % data)
ruamel.yaml.representer.RepresenterError: cannot represent an object: <YAMLMacros.lib.extend.Prepend object at 0x11031e190>

I've been dogfooding this syntax file for the last 1 week working heavily with vue syntax files and I definitely think it's good enough for a first release.

skyronic · 2017-11-28T13:30:57Z

@Thom1729 can you make a gist or paste here the yaml that's generated from the macros file? Also is there a way I can run it without sublime's build system? I noticed there's a cli.py file is there a way I can run that and test it out?

Would definitely like to do this the right way but considering that sublime can release the dev build anytime now I'd rather have this be released in a usable manner so people on dev builds can try it out and report any other bugs.

Thom1729 · 2017-11-28T14:39:08Z

Are you building exactly the example I posted, or an improved version thereof? I can reproduce that error if I remove the !extend tag.

skyronic · 2017-11-28T16:17:31Z

@Thom1729 Yes I am using the exact one you pasted with Yaml Macros v2.1.0 on OS X With Sublime text 3154

Thom1729 · 2017-11-28T17:39:01Z

You need a copy of this version of the HTML syntax, which includes fixes to some whitespace issues. The PR hasn't been merged into core yet. I was able to reproduce that error by using the current version of the HTML syntax without those fixes. Sorry for the confusion.

The YAML Macros extend library should probably issue a warning if it tries to !prepend to a nonexistent list. Better error handling is definitely a priority for the next major version.

cedeber · 2017-11-28T23:26:56Z

Hi. Could you also add typescript support please. Just lang="ts" and scope:source.ts. Thanks.

Thom1729 · 2017-12-01T03:17:25Z

@skyronic Is the example working for you now?

skyronic · 2017-12-02T12:02:54Z

@Thom1729 Sorry was bit busy this week couldn't respond. The core of it works really well, thank you so much for taking the time to put this together. Couple of questions

I'm currently working on adding support for other languages. Here's how I'm handling the coffescript:

contexts: !merge
  main: !prepend
    - include: vue-tags
    - include: script-langs

  tag-attributes: !prepend
    - include: vue-directive

  script-langs:
    - match: (<)(script)\b[^>]*(lang)\s*=\s*([\'\"]coffee[\'\"])\s*.*(>)
      captures:
        1: punctuation.definition.tag.begin.html
        2: entity.name.tag.script.html
        3: entity.other.attribute-name.html
        4: string.quoted.double.html
        5: meta.tag.script.end.html
      embed: scope:source.coffee
      escape: (</)(script)(>)
      escape_captures: 
        1: punctuation.definition.tag.begin.html
        2: entity.name.tag.script.html
        3: meta.tag.script.end.html

So the downside is the attributes aren't highlighted

I can possibly use include: tag-attributes to make this work but that didn't work. But this is a lower priority issue I guess.

The other issue is that it's highlighting the HTML outside the template block. This should not be happening normally.

<template>
  <div>
  </div>
</template>

<p></p>
^^^^ there shouldn't be HTML highlighted outside the template block

<script>
  
</script>

Thom1729 · 2017-12-02T18:18:15Z

Consider this excerpt from the HTML syntax:

- match: '(<)((?i:script))\b(?![^>]*/>)(?![^>]*(?i:type.?=.?text/((?!javascript).*)))'
      captures:
        0: meta.tag.script.begin.html
        1: punctuation.definition.tag.begin.html
        2: entity.name.tag.script.html
      push:
        - match: (?i)(-->)?\s*(</)(script)(>)
          captures:
            0: meta.tag.script.end.html
            1: comment.block.html punctuation.definition.comment.html
            2: punctuation.definition.tag.begin.html
            3: entity.name.tag.script.html
            4: punctuation.definition.tag.end.html
          pop: true
        - match: '(>)\s*(<!--)?'
          captures:
            1: meta.tag.script.begin.html punctuation.definition.tag.end.html
            2: comment.block.html punctuation.definition.comment.html
          embed: scope:source.js
          embed_scope: source.js.embedded.html
          escape: (?i)(?=(-->)?\s*</script)
        - match: ''
          push:
            - meta_scope: meta.tag.script.begin.html
            - match: '(?=>)'
              pop: true
            - include: tag-attributes

It only consumes the opening <script part of the tag and uses a lookahead to see if the contents are JavaScript. The attributes are handled in an anonymous context that inclues tag-attributes. A similar approach should work for CoffeeScript and for any other languages.

This approach does have some downsides. It can be brittle if there's unexpected whitespace or in other corner cases. The ideal solution would be to have a special set of contexts for handling script tag attributes. You would have one context per language; each context would include tag-attributes but have special handling for attributes like type and lang. If one of these contexts encountered an attribute indicating a different language, it would set to that language-specific script attribute context. This would be a more robust approach, and it would be easy to extend with more languages. (In fact, it might even be possible to automate this using macros.) I have not tested this approach.

The other issue is that it's highlighting the HTML outside the template block. This should not be happening normally.

Ah; apparently I've misunderstood the Vue component file structure. Currently, the best way to remedy this would probably be to have two sublime-syntax files. One would be for HTML augmented with Vue templating stuff, and the other would be for the entire component. The component syntax would include the augmented HTML syntax.

skyronic · 2017-12-09T17:09:38Z

Hey @Thom1729 I've made a lot of changes and incorporated your suggestions and yaml macros. Can you take a quick look? Hoping to get this wrapped up in the weekend 😄

Thom1729

As it stands, the HTML.sublime-syntax file will be picked up by Sublime, possibly interfering with the core HTML syntax or other third-party HTML syntaxes. This is a problem I ran into myself when I created the YAML Macros examples.

One solution is to add a hidden: true to HTML.sublime-syntax. The disadvantage here is that when you replace the file with a newer version from upstream, you have to remember to patch it.

What I'm doing now is simply changing the file extension from .sublime-syntax to .yaml. YAML Macros doesn't care one way or the other. The disadvantage here is that PackageDev's sublime-syntax highlighting won't automatically be applied, but this is a minor annoyance.

Thom1729 · 2017-12-09T19:01:44Z