Add support for odoc

[rizo]

Based on the original discussion in ocaml/dune#895 (comment).

It would be great to have support for odoc in mdx. It is a common practice to include examples in API documentation and those can get out of date very quickly. Additionally the new odoc version will include support for documentation pages which would benefit from this feature.

Here are some questions I have regarding the potential integration:

• How would be the OCaml code blocks matched? Should we find all occurrences of {[ ... ]} in the documentation or is it better to have a special annotation for this (some examples might not be runnable at all on purpose)?
• Does it make sense to support the sh syntax?
• Would it be possible to support Reason syntax?

I would be glad to work on a PR for this feature in the next couple of weeks. Suggestions about how this could be implemented would be very appreciated! :)

Thanks!

[samoht]

I don't know if it's better to add support for .mld directly in mdx or to write a new front-end (in that repo or in a different one -- I'm happy to host it here if it's simpler). A new frontend makes more sense I think, but it can re-use some of the logics in mdx and mdx.top.

I think you will be interested by:

• Mdx.Toplevel.of_lines which translate a list of lines (think a "code block") into a list of commands
• And then Mdx_top.eval which takes a multi-line command and returns its output/result (as a normal toplevel would do).

The code block parser is pretty simple (it's actually just a lexer) and I think for mld you want to use your own lexer/parser anyway. You should also probably write your own top-level expect loop and re-use some of the machinery defined in bin/test/main.ml.

An other interesting option is to add a mld subcommand to mdx to automatically generate mld files from md files -- this could be useful for tools when you want your doc to be in sync with the main README.md.

[gpetiot]

I plan to work on this feature very soon, did you start working on this @rizo ? (is there something I could re-use?)

[rizo]

Hey @gpetiot! Sorry for the late reply. I haven't started working on this feature yet, I'm currently working on more critical odoc features. Feel free to work on this and let me know if you have any questions.

P.S. See the linked discussion for some details about odoc integration.

[rizo]

Update on this: I started working on the code block extraction logic in odoc to allow further integration with mdx. Here's the PR and discussion: ocaml/odoc#303.

[XVilka]

Sadly the PR seems stalled.

[jsomers]

I've written a working version of this. It doesn't support shell, and it doesn't support mdx's notion of "sections". All the other environment/label stuff should work, because all I've done is created a new lexer -- for finding code blocks inside mli docstrings -- and new pretty printers.

So suppose you had this in an mli:

(** This is a doc comment with a code block in it:

    {[ # let x = 1 + 1 ]}
*)

The new mdx, when passed the flag --syntax=mli, is able to replace this with:

(** This is a doc comment with a code block in it:

    {[ 
       # let x = 1 + 1
       val x : int = 2 
    ]}
*)

I will submit a pull request soon, but in the meantime, the high-level approach involves:

• Adding a new variant Syntax.Mli alongside Syntax.Normal and Syntax.Cram.
• Adding a new lexer that, instead of consuming the file line-wise, consumes it character-wise, and extracts the contents of {<metadata>[...]} blocks inside (** ... *) docstrings as Blocks. Everything else stays as Text.
• Threading syntax through to places where it wasn't already, to add slightly different pretty-printing behavior. (In particular we want to process newlines differently, and we process padding within blocks differently, since you might begin a code block already indented.)

Curious to hear feedback on this approach. After some more testing and reviewing, I'll submit a pull request.

[samoht]

That seems like a good approach to me! I'd be very interested to review your patches. Also curious to hear feedback from @jonludlam who is working on the next version of odoc.

[sorawee]

The support for mld and mli are now implemented, so I believe the issue can be closed.