LaTeX Reader does not handle environment delimiters or block commands defined with a macro

[timtylin]

This issue pertains to all versions up to current HEAD. Perhaps the most succinct example is with equations:

Command
pandoc -f latex -t native --mathjax

Input

\newcommand{\BEQ}{\begin{equation}}
\newcommand{\EEQ}{\end{equation}}

\BEQ
y=x^2
\EEQ

\begin{equation}
y=x^2
\end{equation}

output

[Para [Str "y=x",Superscript [Str "2"]]
,Para [Math DisplayMath "y=x^2"]]

Considering that environment delimiters are usually prime candidates for macro shorthand, this is probably a genuine shortcoming. As far as I can tell, block commands have a similar problem.

I can try to fix this, but I'm not sure exactly how you would want to approach the problem. We can either:

1. monkey-patch all the individual parsers (and all controlSeq "end" calls) with a map to applyMacros.
2. bite the bullet and do multi-pass parsing, change readLaTeX to first resolve all macros on the input string, then do the usual parse.

[jgm]

+++ Tim Lin [Sep 13 13 03:33 ]:

I can try to fix this, but I'm not sure exactly how you would want to
approach the problem. We can either:
1. monkey-patch all the individual parsers (and all controlSeq "end"
calls) with a map to applyMacros.
2. bite the bullet and do multi-pass parsing, change readLaTeX to
first resolve all macros on the input string, then do the usual
parse.

Multi-pass parsing might be the cleanest. (I've tried to avoid it in
general, but note that there's already a function to go through the
whole input text and process include files.)

Another option, perhaps a nice middle ground, would be to have a
macro parser in both block and inline (not for macro definitions
but for actual macros), which would resolve the macro and simply
insert the result into the input stream.

Something like

result <- processMacro
inp <- getInput
setInput (result ++ input)
return mempty

[timtylin]

I've made significant progress towards the multi-pass approach.

I first considered the latter suggestion (which requires texmath to export macroParser in order to write a Inlines parser for macro commands), but then realized that the number of edge-cases to handle will quickly escalate if I munge it into the blocks/inlines parsing paradigm. Considering the large number of look-aheads existing in the code already, I don't really feel comfortable going down this path. Hence, I bit the bullet and stripped out all the macro definitions.

Like you said, the scaffoldings are already in-place for the handleIncludes preprocessor, so writing another one isn't too bad.

If you'd like, I'll finish cleaning up some regressions (such as the \pdfstringdefDisableCommands{\renewcommand{\sout}{}} command in the unit tests), then issue a pull request for the feature branch.

[jgm]

Hm. I still don't see the problem with the "apply macro and push the result into the input stream" idea.
You'd need something like this:

processMacro :: Monoid m => LP m
processMacro = try $ do
  name <- anyControlSeq
  guard $ name /= "begin" && name /= "end"
  guard $ not $ isBlockCommand name
  star <- option "" (string "*")
  let name' = name ++ star
  rawargs <- withRaw (skipopts *> option "" dimenarg *> many braced)
  let rawcommand = '\\' : name ++ star ++ snd rawargs
  transformed <- applyMacros' rawcommand
  if ('\\':name) `isPrefixOf` transformed
     then mzero  -- no macro was applied, move on
     else getInput >>= setInput . (transformed ++) >> return mempty

Then, just add processMacro to the beginning of inline, and to the beginning of block. (If we did this, we could get rid of some of the similar code that now exists in the handlers for block and inline commands.)

You'd need to do pretty similar processing anyway in a multipass approach, right?

Ultimately it would be good to try both approaches and benchmark to see which has the best performance.

[timtylin]

I was mainly persuaded to go for the preprocessor approach after realizing that it's a closer conceptual approximation to the expansion step in the actual rendering process used by TeX. My intuition is that future macro edge cases related to TeX quirkiness can fixed easier in this paradigm, and that in general this provides a cleaner conceptual separation. Without it, anyone trying to help fix macro issues would have to spend quite some time groking the entire LaTeX reader parsing model.

Performance-wise, the processMacro parser is likely to be cheaper to run, but to resolve my test case it also has to be also included in verbEnv (except in verbatim envs, etc). Before you know it, the amount of look-aheads may incur performance penalties asymptotically approaching the expansion processor approach. I do hate how hand-wavy this argument sounds, so I'll try to setup a benchmarking test between the two approaches when I have the chance. Particularly troubling for the handleMacros preprocessor is the mass number of single-char tokens added to the end of a list (perhaps a nice chance to break out Data.Sequence?)

I wonder if there's other objections to the macro expansion preprocessor model aside from performance issues?

[timtylin]

I partially take back what I said about using Data.Sequence leading to substantial performance gains. I just realized that Parsec's many (used in many macroToken) is already defined as

many :: ParsecT s u m a -> ParsecT s u m [a]
many p
  = do xs <- manyAccum (:) p
       return (reverse xs)

[jgm]

No objections in principle to the two-pass method; it just seems a bit uglier to me, but you may be right that it's simpler and better.

I think that eventually we should probably put handleMacros right into readLaTeX from Text.Pandoc.LaTeX, rather than applying it in pandoc.hs. (In addition, Text.Pandoc.LaTeX should export something like readLaTeXWithIncludes in the IO monad, so we can get this out of pandoc.hs too.)

On the performance problem with lots of 1-character strings: one simple improvement would be to replace the anyChar with something like many (noneOf "\\%"). It would still be good to know what the performance impact of handleMacros is.

[jgm]

This would also handle #934

[jgm]

Be sure to look at these related macro issues: #3236, #987, #2118, #1390, #2114

[jgm]

One issue with modifying the input stream is that you lose good source position information.
(If the macro adds lines, line numbers of errors will be misleading.)

[jgm]

I've got an idea for how to solve this, keeping source position info.

Instead of parsing a [Char], we parse a stream of tokens (produced by a tokenizer).

data Tok = Tok (Line, Column) TokType Text
data TokType = CtrlSeq | Spaces | Newline | Symbol | Word

Then, when we expand a macro, tokenize the expansion and set the source position of each token to the source position of the original unexpanded macro.

This would be a significant rewrite but might have other advantages as well.