C++ code highlighting fails for large files

[ArashPartow]

• VSCode Version: 1.4.0 date: 2016-08-04
• OS Version: Windows 7/10

Upgraded vscode installation, the C++ code highlighting now starts to fail at around line 4000

The following file was used: https://raw.githubusercontent.com/ArashPartow/exprtk/master/exprtk.hpp

The following screenshot demonstrates the problem: http://pasteboard.co/hJ6RUAZtU.png

The following is from the start of the file: http://pasteboard.co/hJ7kaH2X7.png

The following is from the end of the file: http://pasteboard.co/hJ7Gp4nLR.png

---

Tried other files with the same results:

• https://github.com/ArashPartow/strtk/blob/master/strtk.hpp
• https://github.com/ArashPartow/wykobi/blob/master/wykobi.inl

[ramya-rao-a]

@ArashPartow Are you using any extensions for C++ or just VS Code out of the box?

[ArashPartow]

@ramya-rao-a I'm using the C/C++ for Visual Studio Code extension.

http://pasteboard.co/3A56demY.png

I have tried with the extension being active and disabled and still get the same results. Though I did notice that when the extension is disabled the whitening seems to happen a couple of thousand lines later than when it is enabled.

[aeschli]

@alexandrudima The textmate tokenizer seems to return tokens, but we don't show them.

[alexdima]

@aeschli I have improved vscode-textmate's npm run inspect to allow for loading multiple grammars. When both the C++ and C grammar are loaded, at line 17916 for this file, the rule stack grows to 101 and tokenization stops. ( the commit is microsoft/vscode-textmate@5aadb4e ). The relevant command line is:

C:\Alex\src\vscode-textmate>npm run inspect "..\vscode\extensions\cpp\syntaxes\c++.json" "..\vscode\extensions\cpp\syntaxes\c.json" ..\..\working_dir\11597\exprtk.hpp >out.txt

I will spend some time to isolate the constructs that lead to rules being pushed without being popped by the grammar. Once I do so, where can I file the bug (where do these grammars come from?)

[aeschli]

The grammars are from https://github.com/atom/language-c. You can see this by checking the OSSREADME in the corresponding extension.

[alexdima]

Thank you, I asked because I saw two entries. Is the textmate/c.tmbundle still used?

• https://github.com/atom/language-c
• https://github.com/textmate/c.tmbundle

[aeschli]

We still have /extensions/cpp/syntaxes/Platform.tmLanguage in there from the original https://github.com/textmate/c.tmbundle as it is included by other languages (Objective-C)

[ArashPartow]

Update: I've recently updated to version 1.6.1 and am still seeing the above described problem.

[ArashPartow]

Update: I've recently updated to version 1.7.0 and am still seeing the above described problem.

http://imgur.com/a/tCSKk

[ArashPartow]

Update: I've recently updated to version 1.7.1 and am still seeing the above described problem.

[ArashPartow]

Update: I've recently updated to version 1.7.2 and am still seeing the above described problem.

[ArashPartow]

Update: I've recently updated to version 1.8.0 and am still seeing the above described problem.

[ArashPartow]

Update: I've recently updated to version 1.8.1 and am still seeing the above described problem.

[ArashPartow]

@alexandrudima @joaomoreno Just tried out the fix using version 1.9.0 and it's all working great.

Thank-you!