Heredoc strings no longer syntax highlighted

[andremleblanc]

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem in Safe Mode: http://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
  • Followed all applicable steps in the debugging guide: http://flight-manual.atom.io/hacking-atom/sections/debugging/
  • Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
  • Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description

Heredocs used to use the syntax highlighting for that language, but it appears that isn't currently working with Tree Sitter Parsers enabled.

Before:

Current:

Steps to Reproduce

1. Have Tree Sitter Parsers enabled.
2. Use heredoc syntax in Ruby grammar.

Expected behavior:

It highlights the content of the heredoc in that language's syntax.

Actual behavior:

No syntax highlighting.

Reproduces how often:

Consistently.

Additional Information:

Running on macOS Mojave.

Versions

1.32.1
1.33.0-beta1

[rsese]

Thanks @andremleblanc! Reproduced on macOS 10.12.6 with 1.34.0-nightly10.

[Michael-Ludgate]

The bug report contains a clause, differing from the title, that I believe makes it more specific than the problem is. Apologies if this has been tested and found to be specific in your case.

I'm using HEREDOC (tilde, though dash is the same) with Ruby syntax, as reported, however I don't have tree-sitter.

Also noticed that if the ending HEREDOC is typed, at the correct indentation level, after the message (as opposed to the likely more standard practice of typing opening/closing HEREDOC and populating) the syntax parser fails to match the closing HEREDOC, leaving it colored grey.

[JoshCheek]

I'm hitting both issues:

• Syntax highlighting doesn't happen in a heredoc anymore
• I can reproduce @Michael-Ludgate's issue, too (it's like it doesn't realize it's in the string), but only when I use certain delimiters. Eg it doesn't happen if I use a delimiter of HERE, or RUBY, but it does happen if I use a delimiter of HEREDOC or ABCD.

Anyway, the syntax highlighting is a huge feature for me, depending on what I'm doing, I might go to Atom just b/c of this feature (eg when I'm playing with SQL).

[JoshCheek]

Oh, that second one is interesting, it seems to be related to the letter C.

Put this into your editor, select the ABCD and do Cmd+d to highlight the second one. Then slowly type ABCD, it's highlighted as a string for AB, but then loses the highlighting on the C. If you skip C and go to D, it remains highlighted correctly. We can see the same thing in @Michael-Ludgate's example, the word HEREDOC has a C in it, and it correctly highlights as a string up until that letter is typed. If you change it to HEREDOK, it doesn't get messed up.

def m
  <<~ABCD
  msg
  ABCD
end

[JoshCheek]

Oh, okay, so it's going wonky on the highlighting because it applies "C" language syntax highlighting to it. So syntax highlighting does still get applied, but only sometimes. Maybe there's an issue with how it looks up the language? Eg here it sees HEREDOC, notices the C in there, and applies C syntax highlighting, do SQL, and it doesn't recognize it as SQL. Inspecting the DOM shows it's been given a class of syntax--string.

[JoshCheek]

I think this commit is part of the problem. I don't think that change does what it is presumably intended to do. Eg it looks like the author thought it was shorthand for mapping multiple keys to the same value, but instead it created a single key whose name was a conjunction of all the keys. I don't know how tree-sitter works, maybe it treats keys like this as the original, but WRT cson, the new syntax is not translated to the old syntax. Here, I convert it to JSON and we see that the key is not broken out into multiple key/value pairs:

[JoshCheek]

Followed the instructions to make changes locally (here to put atom into dev mode, and here to link local changes into the dev-mode). Ran a git-bisect on it.

This is the commit where it breaks.

Here's the output of the git bisect (in another window I was restarting atom each time I did this and looking to see if it would highlight SQL correctly):

🐠  git bisect start

🐠  git bisect bad

🐠  git bisect good 0909a11557725f117dea6f39aa2b08241a887ff2
Bisecting: 31 revisions left to test after this (roughly 5 steps)
[726281fc9d9fe4e9e3e20a61f5912b062625edcb] Merge pull request #236 from atom/mb-compatible-scope-descriptors

🐠  git bisect bad
Bisecting: 14 revisions left to test after this (roughly 4 steps)
[e5a8d8e66728c013e1a009bcdb2facfd044270f9] Merge pull request #225 from atom/mb-tree-sitter-parser

🐠  git bisect bad
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[701ad5d744e23090655f1c2e725f18cfe6f4ae7c] Add tree-sitter Ruby grammar

🐠  git bisect bad
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[73ec57162fb7eb0035dedc0026911e3d2a8aee4b] :racehorse: Improve the regexp pattern

🐠  git bisect good
Bisecting: 2 revisions left to test after this (roughly 1 step)
[ef4eb5691842a369b83f6465defaad8b82ad191a] Merge pull request #215 from stevenpetryk/variables-ending-in-do-are-not-blocks

🐠  git bisect good
Bisecting: 0 revisions left to test after this (roughly 1 step)
[d88a4cfb32876295ec5b090a2994957e62130f4a] Prepare 0.71.4 release

🐠  git bisect good
701ad5d744e23090655f1c2e725f18cfe6f4ae7c is the first bad commit
commit 701ad5d744e23090655f1c2e725f18cfe6f4ae7c
Author: Max Brunsfeld <maxbrunsfeld@gmail.com>
Date:   Fri Dec 8 09:28:08 2017 -0800

    Add tree-sitter Ruby grammar

:040000 040000 32ce74581b8f1777099d63e11f36f2625814dd82 adcea68aa1e140b88926e33c0f3849582360dce9 M	grammars
:100644 100644 0db247528eaa512b24d4a033c36835c164453f75 c88b06b4fee6a4c54319b0c8b1ae1107037a249a M	package.json

---

Additionally, I put print statements into the injection point language registration (line 6):

language-ruby/lib/main.js

Lines 4 to 8 in ea3824d

	atom.grammars.addInjectionPoint('source.ruby', {
	type: 'heredoc_body',
	language (node) { return node.lastChild.text },
	content (node) { return node }
	})

For <<~HEREDOC, it was returning the language of "HEREDOC", but for "SQL", it was returning an empty string (or possibly something that prints like an empty string).

[JoshCheek]

There are apparently multiple grammars which are loaded, and the one that is used is determined by a configuration value. This package uses this feature to test against the old parser (textmate's?), which means that the tree-sitter stuff doesn't get tested by this lib.

On the bright side, it reveals an easy solution:

atom.config.set('core.useTreeSitterParsers', false)

If you enter it into the debug console (cmd-opt-i) it will immediately update to be correct, and will persist this change in Atom's config.cson.

[BerkhanBerkdemir]

I can reproduce this issue with Atom 1.35.1 and Ubuntu 18.04; however, #249 (comment) is useful.