Support nested LaTeX macro expansions in Markdown reader

[bwhelm]

With pandoc 2.1 (compiled with pandoc-types 1.17.3, texmath 0.10.1, skylighting 0.5.1) on Mac (10.13.2), I'm having problems with parsing LaTeX commands in header-includes. In general, I'm processing a file of the form:

---
header-includes:
- [LATEX COMMAND]
---

Testing.

through pandoc -standalone -t native. The problem takes many forms, depending on what replaces [LATEX COMMAND].

1. With [LATEX COMMAND] replaced by \command{\first}[second]{\third}, I end up getting

    Pandoc (Meta {unMeta = fromList [("header-includes",MetaList [MetaInlines [RawInline (Format "tex") "\\command{\\first}",Str "[second]{",RawInline (Format "tex") "\\third",Str "}"]])]}) [Para [Str "Testing."]]
   

   Note that everything after {\first} is interpreted as a string, resulting in escaped square brackets (\[ and \]) in the LaTeX output. Clearly that's not desired.

2. Searching around, I see similar problems reported and solved. Here's one apparently working example: https://groups.google.com/d/msg/pandoc-discuss/C4WABPg3GWs/cKLiKOAqBQAJ. This doesn't work for me. Instead, using:

    - \let\oldsection\section
    - \renewcommand{\section}[1]{\clearpage\oldsection{#1}}
   

   for header-includes, I end up getting:

    Pandoc (Meta {unMeta = fromList [("header-includes",MetaList [MetaBlocks [],MetaBlocks []]),("title",MetaInlines [Str "Minimal",Space,Str "Working",Space,Str "Example"])]})
   

   That is, the content of the header-includes simply goes missing.

3. In that same thread, I see the following advice: https://groups.google.com/d/msg/pandoc-discuss/C4WABPg3GWs/4Ij4WlCqBgAJ. In short, enclose raw LaTeX in \ ... `{=latex}`, as follows:

    - `\let\oldsection\section`{=latex}
   

   (That advice was for the then dev version of pandoc; I don't know whether it's intended to still work.) When I try this, pandoc reports an error:

    [WARNING] Could not parse YAML metadata at line 4 column 3: found character that cannot start any token
    Pandoc (Meta {unMeta = fromList []})
   

4. When I use the trick of defining \noop that had worked before pandoc 2.0, I find simple cases (such as in (1-3) above) work, but the case I'm interested in results in pandoc being stuck in an endless loop. That case is something of the form:

    - \newcommand\noop[1]{#1}
    - \noop{\one[two]{#1}}
   

   The key here is the #1 in the second line: without it, things work fine.

I hope all of that is clear enough. Many thanks for this amazing program.

[jgm]

Questions like this should go to the pandoc-discuss mailing list rather than here. That way we can sort out what's a bug, and what's a question with a simple answer, and we get focused bug reports rather than big lists of possibly unrelated things.

On (1), pandoc has to guess what is included in the LaTeX command. This is a fairly unusual form, with a bracketed optional argument after a regular braced argument, so pandoc guesses wrong. The recommended way of dealing with this is the \noop macro trick. However, you'll probably also have to deactivate macro expansion: pandoc -f markdown-latex_macros.

The -latex_macroswill also fix (2). (See the discussion in #4246, and feel free to chime in there.)

With (3), you're simply running into YAML escaping issues. The easiest solution is to use this form:

- |
     `\let\oldsection\section`{=latex}

but you could also do

- '`\let\oldsection\section`{=latex}'

(4) seems to be a bug. At least, we shouldn't in any circumstances go into an infinite loop. IS this a real example? Why would you use a #1 outside of a macro definition?

[jgm]

I'd suggest you close this issue and open a new issue focused on (4).

[jgm]

Actually, I've already got a fix for (4), so I'll close this.

[bwhelm]

As I was just about to submit a separate bug report. (By the way, the real example comes from \defbibheading in biblatex: see the biblatex manual, p. 87.)

[jgm]

Oh, I see -- you're trying to wrap \defbibheading, which is itself a macro-defining command, inside the \noop macro. AS far as I can see, this just doesn't work in LaTeX: example

\newcommand{\foo}[1]{#1}
\foo{\newcommand{\bar}[1]{\emph{#1}}}
\bar{hi}

This gives

! LaTeX Error: Command \bar already defined.
               Or name \end... illegal, see p.192 of the manual.

I'm not sure how we should treat such cases in pandoc.

[jgm]

OK, here's a case that works in latex:

\newcommand{\noop}[1]{#1}
\noop{\newcommand{\foo}[1]{#1}}
\foo{hi}

Output is "hi".

[bwhelm]

I'm not sure what the general policy should be for pandoc. For my purposes, it works using:

'` ... `{=latex}'

Since this kind of question seems to keep coming up, perhaps this trick should be documented in the extension-yaml_metadata_block section of the manual.

[jgm]

I'll do that.

[jgm]

I've fixed the underlying issue in the LaTeX reader, but it's still not working properly for LaTeX in markdown, so I'm leaving this issue open until we can handle:

% pandoc -f markdown -t native
\newcommand{\noop}[1]{#1}
\noop{\newcommand{\foo}[1]{#1}}
\foo{hi}
^D
[RawBlock (Format "latex") "hi"]

[stroobandt]

Possible workaround