LaTeX syntax highlighting broken with minted and escapeinside

[alexr00]

When using minted with LaTeX with its escapeinside feature (see pages 22/23 in the docs), this breaks VS Code's syntax highlighting, and the rest of the file is (mostly) rendered in red.

A minimal example to reproduce the issue

\documentclass{article}
\usepackage{minted}

\begin{document}
\begin{minted}[escapeinside=||]{python}
    print(|\emph{test}|)
\end{minted}

\textbf{\textrightarrow{} Syntax highlighting broken for the rest of the file!}
\end{document}

Outcome on stock config without extensions:

• VS Code Version: Code 1.77.1 (b7886d7461186a5eac768481578c1d7ca80e2d21, 2023-04-04T23:20:37.202Z); 1.77.3 (704ed70d4fd1c6bd6342c436f1ede30d1cff4710)
• OS Version: Linux x64 6.2.11-arch1-1
• LaTeX-Workshop version if you use it:

Originally from @The-Compiler in microsoft/vscode#180692

[The-Compiler]

Also moving this comment over as it hopefully provides additional context:

---

The "Inspect editor tokens and scopes" feature reveals that VS Code still assumes Python inside the escape:

and then gets stuck in that even past the \end{minted}:

---

Personally I wouldn't mind much if the syntax highlighting for LaTeX inside the |...| was broken somehow (though that being LaTeX again would be a nice bonus, of course). But the rest if the file being all red (with no way to add some clever comment or something to make it "re-sync", at least I couldn't find anything) makes it very hard to edit things unfortunately. Agreed however that this seems rather tricky.

[jlelong]

This issue arises because the Python grammar finds some invalid syntax and gets stuck inside a function call rule. To manually get out of the trap add

\if 0
)
\end{minted}
\fi

I am afraid we cannot do much to solve the root cause of the issue. It would basically require to duplicate all possibly embedded grammars to add a bail out rule. See the discussion in James-Yu/LaTeX-Workshop#1689 for the C++ grammar. This is of course far beyond the scope of this basic vscode built-in extension to provide bailed out grammars and language definitions for over a dozen external languages.

@alexr00 A smart solution would be that VS Code implements a bail out rule when embedding a language inside an other grammar.

[alexr00]

@jlelong that makes sense, I didn't think that this issue would be solvable from the latex grammar. However, I don't think we would know when to do the bail out. Sometimes users do write code that should be scoped as invalid or illegal, so we cannot know when to bail out. I'm not sure this issue can be nicely solved with textmate.

[jlelong]

@alexr00 What I had in mind is the ability to define a bail out pattern when embedding a language. When this pattern is encountered, we leave the embedded grammar and we go back to the calling one.

[alexr00]

@jlelong would this be something that you would define in your textmate grammar at the point that you embed the grammar?

[The-Compiler]

@jlelong Thanks for that manual escape, that's a life-saver for me! ✨

Here was a particularly spicy one:

    \begin{minted}[autogobble,escapeinside=||]{python}
        @pytest.mark.parametrize("a, b, op, expected", [
            pytest.param(1, 2, '+', 3, |\highlightbox{id=}|"add"),
            pytest.param(3, 1, '-', 5, |\highlightbox{id=}|"sub"),
            # ... some more lines ...
        ])
        def test_calc(a, b, op, expected):
            assert calc(a, b, op) == expected
    \end{minted}

    % VS Code workaround
    \if 0
    )))))])
    \end{minted}\fi

[jlelong]

@jlelong would this be something that you would define in your textmate grammar at the point that you embed the grammar?

@alexr00 yes exactly. For instance, embedding the Python language typically looks like

{
	"begin": "some_begin_pattern",
	"end": "some_end_pattern",
	"contentName": "source.python",
	"patterns": [
		{
			"include": "source.python"
		}
	]
}

It would be nice if the top level grammar could escape from the Python one when the end pattern is reached. I originally thought a new pattern was needed to trigger the bail out but I believe it will always be the same as the end pattern.

Currently, we provide a modified C++ grammar which includes the bail out pattern. This approach would make it useless. Embedding a grammar would much more robust.

[alexr00]

@jlelong I have opened microsoft/vscode-textmate#207 for this. Feel free to close and we can continue discussion there.