Text.Pandoc.Pretty: excessive CPU and memory usage

[t-8ch]

The file found under https://gist.github.com/t-8ch/76d21fc1321880f48eb6 triggers excessive resource consumption when converting from html to markdown/plain.
Converting to native format is reasonably fast (0.5 seconds), but plain takes over 5 gigs of RAM and several minutes to convert.

Happens with 704cfc1 (latest master as of now)

Command to reproduce $ pandoc -f html -t plain bad.html

[mpickering]

Here are the results of profiling.

--- Updates

Fusing the definition of realLength helps significantly.

That being said, I'm not getting as bad performance as you are encountering. Are you compiling with -O2?

[t-8ch]

Thanks for looking into this!

I just did cabal configure && cabal build

[mpickering]

The output file is over 900 000 lines long, surely this is indicative of another problem?

I think the problem is that the source file uses lots of nested tables to control the layout. Pandoc doesn't handle this sort of input very well so I suspect that this is the problem.

[t-8ch]

Yes, this input is definitively totally broken. (HTML emails, yeah).
On my system even with your optimization the OOM killer kicks in.
Now I simply limit the memory size of the haskell RTS and open those mails in firefox.

Thanks for looking into this @mpickering!

[jgm]

@mpickering, should this really be closed? It doesn't sound like the problem is really resolved.

[mpickering]

I think the input is pathological but there might be something better we can do when there are lots of nested tables.

[jgm]

The profiling report suggests that the problem is entirely in Text.Pandoc.Pretty.