Design a system for embedding diagrams code within Haddock documentation

[byorgey]

The idea is to be able to write (specially marked) diagrams code within Haddock documentation, then run some sort of preprocessor over it (using diagrams-builder) which will compile the code and insert URLs referencing the images (Haddock supports <<URL>> syntax for embedded images). The images can be hosted on projects.haskell.org. The final result will be Haddock documentation with embedded images next to the code used to generate them -- these can serve as simple examples as well as illustrations of whatever is being explained in the documentation.

The tricky part is that this preprocessor needs to be idempotent. It's no good to replace the code with image URLs, because we don't want to be forced to maintain two separate versions of everything (one with diagrams code and the other with image URLs).

One idea is to just have the preprocessor look for comment sections of the form

-- <<URL>>
--
-- > foo = circle 3 # red
-- > example = foo ||| foo

that is, if you want the preprocessor to render some diagrams code, you must prefix it with a <<URL>> link (presumably the link could be garbage or empty initially). The preprocessor will then update the URL if necessary -- diagrams-builder already has a facility for computing a hash of the code to see whether it needs to be rebuilt.

[byorgey]

To clarify, the goal of this isn't just to get something that works for diagrams' own documentation, but a usable tool for anyone to enhance their Haddock documentation with diagrams. This fits into a larger strategy of coming up with ways to embed diagrams in various sorts of documents, which I think could be a "killer feature" for diagrams. See also this blog post (about embedding in blog posts and LaTeX documents).

[byorgey]

I started looking into this a bit today. Technically I think there is nothing too hard lurking here. We can use haskell-src-exts to parse code including comments, modify the comments, and exact-print the file back out. diagrams-builder gives us the tools we need for actually building the diagrams.

The hardest, part, I think, is in designing what it should actually do! The requirements are something like

1. The ability to write diagrams code inline and mark it somehow so the diagrams-haddock tool notices it
2. The ability to either include such diagrams code in the Haddock output or not (when documenting a particular library you may just want a picture, and not to show the diagrams code, especially if it is a library that itself has nothing to do with diagrams)
3. Have a way to insert image URLs in Haddock comments, mark them for processing by diagrams-haddock, and link them to a specific piece of diagrams code
4. The tool must be idempotent!

It's not obvious (to me at least) what the best way is to satisfy all these requirements, or if it's even possible. Requiring idempotency hurts a lot, because of how restrictive Haddock is as a markup format.

[nomeata]

For 3 and 4: It seems that data urls “support” # parameters, i.e. they are ignored. So the URLs could be marked up as

<<<url:text/svg+xml;base64,DummyThatThisToolMayReplace#diagram:foo>>>

and now this url is recognizable by the tool as where the final diagram should go to, and putting the diagram in there is idempotent, as the #diagram:foo part can stay there.

For 2: If the code will be found in non-haddock-comments as well, then this is easy.

And for 1, the most simple solution might be that if there is a data url referencing #diagram:foo, then the tool will look for any binding foo = ....

If you don’t like putting those that should not show up in the haddock documentation in real comments: I believe you can have haddock comments that are not shown by naming them, e.g.

{-$foo
> ...
-}

but not referencing this named chunk from the exports list of the module.

[byorgey]

Ah, interesting idea! I hadn't really been thinking in terms of data URLs before, but that does nicely solve 3. The only downside (not insignificant!) is that source files will now contain large blocks of base64-encoded gunk which one has to wade through when editing the code. Ah, but it looks like <img src="URL#diagram:foo"> "works" too, so that solves 3 independently of using data URLs vs hosting the images elsewhere.

I agree re: 2. I think the only difficulty with 1 is when one does want to include the code in the Haddock output; presumably we don't want the "markers" identifying and naming the diagrams code to also be output. But perhaps there's no way around that and it's not really that bad to have something like (say)

-- | Here is an example of a square
--
-- <<dummy#diagram:foo>>
--
-- > -- [diagram: foo]
-- > dia = square 1

where -- [diagram: foo] actually shows up in the typeset code.

[nomeata]

But, why do you need markers? Instead of

-- > -- [diagram: foo]
-- > dia = square 1

just write

-- > foo = square 1

The presence of <<dummy#diagram:foo>> is enough to trigger the foo = ... line to be taken into consideration.

[byorgey]

Aha! Indeed, I think that will work quite nicely.

[mgsloan]

This would be very cool!

(btw, congrats on getting the release out!! Good stuff! Hoping to get time
later this week to release the toy framework)

You might want to take a look at the source code of
https://github.com/divipp/ActiveHs , as it does something very similar to
this. In particular, it supports a diagrams-like library:
http://pnyf.inf.elte.hu:8000/Diagrams_en.xml

-Michael

On Sun, Dec 16, 2012 at 6:11 PM, Brent Yorgey notifications@github.comwrote:

Aha! Indeed, I think that will work quite nicely.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-11428420.

[nomeata]

Also, I wonder if there is not something even more general to aim for: TH slices in haddock documentation? After all, I might not only want to generate a diagram but also show the textual output of some calculation in the documentation, or even some marked up calculation.

So if haddock would allow TH splices (as much as I don’t like TH) in comments, maybe of type "Q String" or "Q HaddockSyntaxAST" the issue at hand becomes a simple library function to use, with no additional preprocessor needed.

But then, better find a solution that works for diagrams first and then think about the silver bullet :-)

[byorgey]

A couple of relevant links:

• John W recently added an extra-html-files field to .cabal files, the contents of which will get bundled with the package tarball and installed along with the documentation: Add extra-html-files, for installing extra html files haskell/cabal#1182 . This means no longer having to choose between hosting generated resource files elsewhere or sticking them in as data URLs.
• Interestingly, the split-channel package illustrates another approach which works in the meantime (until a new version of cabal is released and in use on Hackage, which might be a while!), using Cabal hooks to take responsibility for copying extra resource files itself. Here's the contents of split-channel's Setup.hs:

import Distribution.Simple
import Distribution.Simple.Setup   ( haddockDistPref, Flag(..))
import Distribution.Verbosity      ( normal )
import Distribution.Simple.Utils   ( copyFiles )
import Distribution.Text           ( display )
import System.FilePath ((</>))
import System.Directory

-- Ugly hack, logic copied from Distribution.Simple.Haddock
haddockOutputDir flags pkg = destDir
   where
     baseDir = case haddockDistPref flags of
                      NoFlag -> "."
                      Flag x -> x
     destDir = baseDir </> "doc" </> "html" </> display (packageName pkg)

main = defaultMainWithHooks simpleUserHooks {
    postHaddock = \args flags pkg lbi -> do
        copyFiles normal (haddockOutputDir flags pkg) [("doc","split.png")]
        postHaddock simpleUserHooks args flags pkg lbi
  }

[byorgey]

I've begun working on something putting all the above discussion together; it's in the repository here:

http://github.com/diagrams/diagrams-haddock

At this point it seems to really work -- it just needs a nicer API, more documentation, and so on. So I'm going to go ahead and close this ticket.