Generate documentation for packages again

[teepeemm]

This PR generates the four files blib/lib/LaTeXML/texmf/(latexml|lxRDFa).sty(.ltxml)? from blib/lib/LaTeXML/texmf/(latexml|lxRDFa).pm by using blib/lib/LaTeXML/texmf/sty_ltxml.PL. I have adjusted Makefile.PL so that this happens during the make step of installation (after things get copied over to blib). The make step also creates blib/man3/LaTeXML//texmf//latexml.3 and blib/man3/LaTeXML//texmf//lxRDFa.3.

I have also adjusted genpods so that it creates tex versions of the documentation, and manual.tex will include the documentation in the new appendix chapter B, resolving issue #844. (If we want to have it in the actual manual, we could put latexml as subsection 4.0.1 and lxRDFa as 8.1.1.)

Because the new pm is a mixture of the old sty and ltxml, I don't see a good way to diff them between the versions. The best I can come up with is to generate the new sty and ltxml and diff that against the old (and | grep -v '> %' or | grep -v '> #' to ignore the comments in the old versions). Eventually, you'll see that they're identical.

Todo:

1. Fix any mistakes in my interpretation of the comments/documentation
2. During make install, we should also install the documentation (assuming they have a TeX installation). From https://tex.stackexchange.com/a/146196/107497, it appears we would want to copy the documentation to $TEXMF/doc/latex/latexml/. But I'm not sure what we would want to use:
   1. If they have Pod::LaTeX, then we can use the tex that genpods creates to create pdf files in doc/manual/texdoc (and optionally the xml/html)
   2. Or we could include those tex (or pdf) files in the git distribution
   3. Otherwise, we might be able to use the blib/man3/LaTeXML//texmf//*.3 files (I don't know if texdoc can use these)
   4. Otherwise, we should convert the pm files to txt

(This is a reincarnation of PR #1871, which I messed up.)

[brucemiller]

I love what you're trying to do, but maybe not so much in how. Personally, I despise POD, and have long pondered a more TeX friendly, "literate" style, interleaving specially marked comments with code, which could be "trivially" be extracted for documentation. The same approach (with different markers) could be used to embed the .sty version of the code. This would have the benefit of keeping everything together so that the code(s) and documentation are more likely to match (one of the worst problems of POD at the end).

OTOH, @dginev is always warning me against roll-your-own documentation systems; and I haven't found time to do a proof-of-concept (or documention, for that matter!). But then @dginev seems to be disliking TeX itself, so that probably disqualifies his opinions :>

[dginev]

But then @dginev seems to be disliking TeX itself, so that probably disqualifies his opinions :>

gasp

pity, though on occasion I find myself in good company.

FWIW, I am really fond of the way Rust has structured its documentation with rustdoc (even adding optional code snippets that double as examples+tests). Of course we can't have that with POD, as-is.

[teepeemm]

But it's probably better to aim for a Perl friendly style as opposed to a TeX friendly style, since most of what we're doing is Perl. It's possible to interleave POD and not put it all at the end; the only cost is a lot more =cut tags. That's what I've done here with the .pm files; the other files are supporting and slight consequential changes (and I agree with avoiding leaving all of the code at the end). So I'm not quite understanding your objection.

I think it would be possible to create a POD tag that would allow code snippets to become tests. At the moment, though, I can only think of writing the code into a temporary file or running the code through eval. Either way would be vulnerable to someone leaving something like rm -rf / in a code snippet, so I'm not sure about the security implications.

[dginev]

@teepeemm

But it's probably better to aim for a Perl friendly style as opposed to a TeX friendly style, since most of what we're doing is Perl.

You may be amused to find out that back in 2008 I spent some time undoing the opposite approach in a project that was doing latexml bindings - that project used .dtx files to interleave literal documentation, .sty code, as well as .ltxml perl code. And it was a complete nightmare to manage or maintain.

I suspect if you start interleaving .sty code into .pm files you will be one of a handful of people able (or worse - willing) to maintain the combination. It's just not palatable to developers who haven't fully bought into "POD is really worth investing time and effort in".

My own journey through life has left me preferring disentangling the formats at risk of divergence - standalone .sty and .ltxml files. And a pause/punt on how best to document them.

I am currently trying (and until 0.8.7 gets released, likely failing) to spend most of my time rewriting a big chunk of latexml (definitely all of the Perl code) into Rust, something I started back in 2016, and have spent little drops of time in since. My intention is (and I think I am getting more support recently) to seriously increase the percentage of my own time that I spend in that effort, which will include porting any documentation we may have to rustdoc, likely moving it to the inline variant, which tends to be easier to remember to update when the content of a function changes.

Luckily my opinions here are currently disqualified (tongue-in-cheek), and as usual all you need is to convince Bruce. But I thought I'd share which part of the fence I'm sitting on.

[brucemiller]

My personal coding style is not necessarily the Current Fashion: Vertical space is precious, I want to see an entire function and its documentation on the screen at once. I find it silly to waste whole lines for "begin", "{", "}", and so on, and then lots of blank lines besides. Proper indentation (and a decent editor) can make the nesting perfectly clear. POD takes that silliness to another level, especially when you try to interleave; I can't even see the whole documentation. So, really POD a dead end for me [but see below].

What is nice about rust's system is that, unlike POD, it can get a snapshot of API's & function signatures and such (I think?), and that it can integrate with the rust tool chain. But not so much with external tools & materials, eg. to create an entire documentation or website or... I did manage to extract some latex-like material from PODs to generate the current LaTeXML documentation, but it's quite brittle and I neither want to extend that, nor repeat it with a different system.
OTOH, a recognizable style of block comment can be extracted from sources of any programming language (perl, rust,...) to be documentation written in any other (TeX, markdown, pod...).

Moreover, I'm working on a system that deals with a lot of flexibility and expressiveness. Consequently I have to suffer all of TeX's complexities. It would be too cruel an irony to also have to suffer markdown's lack of flexibility and expressiveness just to document the system.

I could maybe/almost live with pod-like block comments such as

=begin documentation
Sometimes $E=mc^2$, but not always.
=end documentation
=begin sty
\def\einstein{Something Wise}
=end sty

if it weren't for all the infernal required blank lines. And then, only if the "sty" parts were embedded in (& extracted from) the latexml.sty.ltxml file itself.

[teepeemm]

@dginev

I am currently ... rewriting ... all of the Perl code into Rust.

Does this mean that we're aiming to convert the whole project over to Rust (except for the TeX and XSLT), or is Perl -> Rust more for your personal edification? Or are you able to have Rust hiding beneath the surface, and LaTeXML will still look like it's mostly written in Perl?

I don't think any of us are great fans of POD. But it seems to be the best that we can do if the documentation is in the file. Unless we're switching to Rust, or we decide to roll our own documentation (in which I agree with Deyan that we shouldn't do).

Should I redo this and have the documentation solely in the .sty.ltxml, and leave the .sty alone? Or are you leaning more towards having the documentation completely separate? I would prefer fewer files, even if it means more infernal blank lines, but am willing to be outvoted on the issue.

[dginev]

Does this mean that we're aiming to convert the whole project over to Rust

Yes, but only if we succeed :> I think Bruce is currently in a neutral reviewer stance about the idea, though I may have gotten him to lean to the "pro" side more heavily recently.

That directly means "no perl" in the new rewrite.

It's both easier and harder than it sounds.

[brucemiller]

It's both easier and harder than it sounds.

A good summary... and a good tagline! Applies to both the Rust rewrite, about which I'm cautiously optimistic. Or even fatalisticly so: it has to succeed, since the Perl approach can't last for all that long. And to the documentation issue. I think that embedded documentation is necessary, else it drifts and is worse then none. I don't see why roll-your-own has to be bad or wrong or hard, just because dtx is. LaTeXML's current system is half hand rolled, but based on POD, which is the source of 90% of it's problems. The worst thing is Perl has no built-in, palatable, block comments. Of course, the proof would be in the pudding of a proof of concept...

[teepeemm]

I think subclassing from POD isn't too far into the "rolling our own" territory, as long as the subclassing isn't too much. Getting the sty or tex out of sty.ltxml only takes a dozen lines of code each, so I think it would stay fairly maintainable. We could also use the standard Pod::Html or Pod::Simple::HTML to get html documentation that texdoc would be able to find (we would need to PREREQ_PM the non-standard Pod::LaTeX to get tex documentation). On the other hand, if Rust is imminent, then it could make sense to wait for that before adding to the documentation and build chain.

So what would we want for documentation for the packages? It would be possible to have the documentation and/or sty reside in the sty.ltxml files, at the cost of many blank lines (this is what the pm files of this pull request actually are). We could also split the sty files out on their own, reducing the blank lines, but risking drifting (the same could be said of the documentation). Or we could wait for an imminent Rust, at which point documentation should become easier. I'm happy with any of those options, but I'm not sure what we're wanting to aim for.

[teepeemm]

My next attempt at this. This PR creates the .sty files out of =for sty POD comments in the .sty.ltxml files. Because we're following POD conventions with =for sty (and =begin sty ... =end sty), it only take a dozen lines of code to extract those comments into the sty files (but does mean that the POD has lots of blank lines). I would argue that makes this approach easily maintainable without having rolled our own documentation approach. (I think this approach would also let us define \LaTeXMLversion and \LaTeXMLrevision in the sty files, but that would require more code and undo what I've just said.)

In addition to the .sty files, make will also create two .html files using make_package.PL. If make then uses install_TeXStyles to copy the html files to $(INSTALLTEXMFBASEDIR)/doc/latexml/, then texdoc will be able to find the documentation. But I was having trouble understanding how that code does its work.