\input in Markdown causes rest of file to be treated as raw LaTeX

[fiapps]

With pandoc 2.2.2, \input in a Markdown file causes everything after the \input to be parsed as raw LaTeX.

echo '\small' > foo.tex
echo 'Markdown parsed *here.*

\input{foo.tex}

*But not here.*' | pandoc -t native

Output:

[Para [Str "Markdown",Space,Str "parsed",Space,Emph [Str "here."]]
,RawBlock (Format "latex") "\\input{foo.tex}\n\n*But not here.*"]

This does not happen with pandoc 2.2.1. It also does not happen if the included file is empty (cp /dev/null foo.tex) or does not exist. It happens even if the included files contains something that pandoc surely knows how to parse (echo '\emph{bar}' > foo.tex).

[jgm]

I understand why this happens now.

withRaw runs the parser, which in the case of \input modifies the token stream. It then tries to identify the next token nxt by looking ahead. This will be the first token in the included file. Then, it tries to identify the original tokens parsed by doing takeWhile (/= nxt) inp, where inp is the original token stream, before modification with the tokens from the input file. Since nxt is not included in this token stream, the whole thing is read.

That's the problem, anyway. The solution is less obvious. It's hard to give a good definition of withRaw that makes sense in a context where commands can change the token stream.

[jgm]

Added test case in issue4781 branch