Annotations/Attributes Scope Naming

[wbond]

Currently the following syntaxes that are part of the default (or will become part of the default) have constructs called annotations, attributes or decorators that modify a function/class/type/etc:

• Java
• Python
• Rust
• C#
• Scala
• Swift

They are not all 100% functionally equivalent, but all seem to generally include:

1. Some sort of punctuation to signal the construct
2. An identifier
3. Optional other distinct tokens including things like parens, strings, integers, etc

Currently the scopes are very inconsistent across syntaxes, and I'd like to get it standardized so users and color scheme authors can benefit. Here are links to some existing discussions about them:

• [Scala] add support for annotations #605
• [Python] decorators highlighting is broken #704
• [C#] Rewriting from scratch #93 (comment)
• Attribute comments were only defined in global scope rust-lang/rust-enhanced#124

My thoughts on the current state of annotations/attributes:

1. While historically they have been highlighted as comments in some syntaxes, I believe this is a disservice to the user since they can contain multiple tokens
2. The entirety of the construct should get meta.annotation or meta.attribute.
3. I think that a consistent punctuation. scope should be applied to the leading symbol
4. The identifier is probably the crux of the discussion. Java currently uses storage.type. The Scala PR has proposed storage.modifier.annotation. Rust is currently scoping as a comment.. Python scopes the @, but just applied a meta. scope to the rest. I forget where C# is right now. We don't yet have Swift in the default packages.
5. All tokens after the identifier should probably get the standard scopes.

Based on this, I think the decisions that need to be made are:

1. Should we use meta.annotation or meta.attribute as the meta scope? It seems C#, Rust, Swift use the term attribute. Java and Scala use annotation. (Python's decorator construct is slightly different, but I feel it probably makes sense to scope it similarly.)
2. What should the scope of the identifier be? I think storage.type is used too much in general. storage.modifier.annotation feels a little too semantically strict to have this be a general-purpose scope name, but has the benefit of being implemented by existing color schemes. variable.annotation would be consistent with variable.function, but the later has always felt a little odd. The other idea is to start using a new top-level scope name of identifier. that allows for more fine-grained control of identifier in various contexts. The downside of this is that no color schemes currently support it.

Pinging the following users for feedback, based on involvement with syntaxes discussed above, or general involvement in this repo:

• @FichteFoll
• @gwenzek
• @djspiewak
• @keith-hall
• @michaelblyons
• @dten
• @Jayflux
• @colinta
• @GregoryBL

[colinta]

My first thought on all this: I hope, at the end, that you make this decision, because I think you're likely the best candidate to see all this at a high level. Very thoughtful points, esp about the identifier scope.

1. Should we use meta.annotation or meta.attribute? Even though I program in Swift all day, I do not find attribute to be an intuitive name for this language feature. I like annotation more, I find it more descriptive, probably because it's a more technical term (anything can be an "attribute", it's such a vague term).
2. What should the scope of the identifier be? I wish I could have an educated opinion here, but I don't quite understand the difference in "meta scope" and "identifier scope" (would love to learn!).

HTH

[wbond]

Yes, I do plan on making the decision, but I want to gather feedback. I realize that changes to scopes do impact users the most, but also color scheme authors. My hope is that as we are making the Sublime Text world a better place, most users will be willing to put up with the hiccups along the way for the long-term benefit. So far it has been a long process, and I’ve learned a ton since early this year. Some choices have been further refined, or changed back after working on more and more syntaxes.

http://www.sublimetext.com/docs/3/scope_naming.html contains guidelines based on everything I’ve learned so far. My intent is to add information from this discussion to that page.

In terms of meta scopes, they are intended to be used to mark up “sections” of code that are composed of multiple tokens. For instance, the entirety of a function definition would be scoped meta.function. Color scheme authors are discouraged from targeting meta.* scopes directly, but ideally they are for refining scope matches in color schemes, or for plugins to locate specific elements of code.

[djspiewak]

I basically agree with all of your major points: the punctuation and meta scopes should be leveraged, and comment is a terrible scope for this sort of stuff. storage.type is absolutely over-used, so I would be strongly opposed to overloading it even further.

I'm not a fan of "attribute", since it feels less descriptive to me than "annotation" or "metadata", but maybe that's my JVM bias showing. storage.annotation or storage.metadata wouldn't be a terrible choice, since most color schemes scope storage. The problem is that most color schemes scope storage.type via storage, and so storage.annotation is practically equivalent to storage.type for most color schemes, and again, I'm not a fan of that.

The class of constructs we're describing here is really generically "metadata", and it is fundamentally different from other classes of constructs and merits its own scoping. So, from a purely clean design space standpoint, the right thing to do would be to open up a new top level scope. meta is already taken, and metadata is too close to that, but annotation seems fine. annotation.pragma (for things like @SuppressWarnings in Java or @BeanProperty in Scala), annotation.doc (for things like @author), and similar. But as you said, pre-existing color schemes aren't going to catch this. Top level scopes shouldn't be introduced lightly.

(as a sidebar, it appears that #pragma is scoped as keyword.control.import in the C/C++ mode, which seems completely out of left field given that most preprocessor directives are quite close in spirit to annotations/attributes)

From the standpoint of "least terrible" relative to the ecosystem, nesting under storage.modifier (still) seems to me to be a good choice for all of the reasons that we chose it on the Scala PR. storage.modifier.annotation or storage.modifier.metadata both seem reasonable. However, if we can get color scheme authors on board with making a change, then introducing a new top-level scope (e.g. annotation) seems like the best way to go.

[FichteFoll]

Disclaimer: The languages of the discussed set that I use(d) are Python (and Java).

1. meta.annotation > meta.attribute. Languages could refine this with meta.annotation.attribute and meta.annotation.decorator, if they wish.

2. Definitely not storage.type. It's kind of over-used but more fundamentally something different semantically.

   variable.annotation/identifier.annotation would work.
   storage.annotation would too, but realistically annotations are rarely affecting storage itself, even though they technically always do in Python, where this would probably make the most sense out of all languages. However, consistency is more important here.

   annotation as a new top-level scope seems overkill for a "small feature" like this that is fundamentally based on arbitrary identifiers, which variable/identifier already solve.

   Thus, my vote goes to variable.annotation, and for Python additionally variable.annotation.function.

Regarding variable vs identifier, I do believe that there is benefit in switching to a more flexible identifer top-level scope, however this would overshoot the discussion in this issue imo. I have a couple remarks regarding this already, so feel free to open a new issue for it, @wbond.

[wbond]

@djspiewak Thanks for your thoughts and time on this. I don't think adding a new high-level scope annotation makes sense. meta is used, but it is supposed to be used for things like covering the whole annotation expression, so we'll definitely use that. The real decision it about the identifier for the annotation, and I've come to a decision about that.

[wbond]

Ok, so I've decided on the following:

1. We are going to use the term "annotations" even when the language calls them an "attribute". This seems less vague, although it obviously doesn't match language terminology in some cases.
2. Annotations should not use the comment scope, unless there is a comment embedded within them, and in that situation the comment scope would only be applied to the actual comment
3. The complete annotation/attribute/decorator expression should be scope meta.annotation. Ideally sub-scopes will be such as meta.annotation.identifier or meta.annotation.parameters. These should not be "stacked", but rather should transition during push/pop/set so that at exactly one meta.annotation* scope should be on the stack at once. You can examples of this in Python or Go function definitions.
4. punctuation.definition.annotation should be applied to any leading symbol(s)
5. The identifier part of the annotation should be scoped variable.annotation. This is consistent with variable.function. It is not great, but it is more likely to be handled by color schemes for now via the variable scope. Syntaxes can feel free, as with other scope, to add further scopes to the end to provide disambiguation.
6. We will discuss the idea of an identifier top-level scope at some later point when we can come up with a migration plan.

Thank you everyone for you time and thoughts on this.

[jrappen]

@wbond Maybe you could add the information from your last post to the scope naming guidelines?

[wbond]

Yes, that is the plan

[wbond]

Just for the record, these will be part of the docs with the next release

[jasonwilliams]

ping @ehuss