Hard to get some symbols inside macro calls and definitions on the user side

[GrayJack]

Not sure if it's a proper bug, but I got myself unable to classify some symbols on Atom inside macros.

Like, when I define something like:

'"+"' : 'keyword.operator.add'

It will classify all + signs, except inside macro invocations and macro definitions

The same are true for all operator-like symbols and the in, but other keywords work just fine.

Looking at the code, I suspect that this happens inside _non_special_token, where in is not there and the operator-like symbols are defined by a regular expression /[/_\-=->,;:::!=?.@*=/='&=#%=^=+<>|~]+/, not separated strings.

Again, not sure if it can be considered a bug or if it should be handled in some way on the user side.

[maxbrunsfeld]

No, I think it's reasonable to change the definition of non_special_token to treat all of the tokens separately. I can't remember exactly why it's currently defined with one single token.

[GrayJack]

Maybe because it is allowed any kind of string, for example I could do mymacro!(1 <--> 10) or some kind of haskell like operator, are all valid in most cases.

But I would like to at least the normal Rust operators be able to be captured inside macros.

[maxbrunsfeld]

In the case of Atom, it should be re-parsing the body of the macro with the Rust parser, in order to highlight it correctly. There aren't well-documented APIs for accessing these nested parse trees, but I think you can use this method to access nodes from 'syntax layers' other than the root layer.

[GrayJack]

I can use this method inside of the Atom tree-sitter grammar definition? And how so?

[maxbrunsfeld]

Oh, I thought you were wanting to access the symbols programatically. For highlighting, I believe that should already work out of the box, because Atom re-parses the contents of token trees as rust source code. This is set up here: https://github.com/atom/atom/blob/0b34be796312a94331acbc4602944cfe3666c76a/packages/language-rust-bundled/lib/main.js#L2-L13.

[GrayJack]

Yeah, true, doing that it does reparse the token tree, but that I would like to happen on token_tree_pattern as well on macro_definition, that is possible?

[resolritter]

This seems to be of similar concern to #98

[amaanq]

They're treated separately now