Document generation system?

[TeroFrondelius]

We need to decide, how to make the manual. I found one potential option:
https://github.com/JuliaLang/JuliaDoc

If I understood correctly this haven't been fully decided in the Julia:
JuliaLang/julia#3988

If you can find more Julia style approach, please bring it to the discussion.

[TeroFrondelius]

Once again others wisdom

Documentation is essential. There needs to be something for people to read, even if it's rudimentary and incomplete. This falls squarely into the "drudgery" category referred to earlier, and is often the first area where a new open source project falls down. Coming up with a mission statement and feature list, choosing a license, summarizing development status—these are all relatively small tasks, which can be definitively completed and usually need not be revisited once done. Documentation, on the other hand, is never really finished, which may be one reason people sometimes delay starting it at all.

[TeroFrondelius]

Here is the Julia style: https://github.com/JuliaLang/julia/tree/master/doc

Julia Documentation README

Julia's documentation is written in reStructuredText, a good reference for which
is the Documenting Python
chapter of the Python Developer's Guide.

Prerequisites for building the documentation

The documentation is built using Sphinx and LaTeX.
On ubuntu, you'll need the following packages installed:

latex-cjk-all
texlive
texlive-lang-cjk
texlive-latex-extra

On OS X, you can install install MacTex using the GUI installer

Building the documentation

Build the documentation by running

$ make helpdb.jl
$ make html
$ make latexpdf

File layout

conf.py Sphinx configuration
helpdb.jl REPL help database
stdlib/ Julia standard library documentation
UNDOCUMENTED.rst Undocumented functions (to be filled in and copied to
the correct location in stdlib/)

Sphinx extensions and theme

The extensions to Sphinx and the theme are in the
https://github.com/JuliaLang/JuliaDoc repository, and can also be used to style
package documentation.

[TeroFrondelius]

@ovainola did you test the @doc? For me it's looks like that in the future the documentation would be clear:http://julia.readthedocs.org/en/latest/manual/documentation/#other-notes

[ovainola]

@TeroFrondelius I tested the @doc macro. There's a bit more boiler to write if you compare with the notation in the link. Though it doesn't hurt if we do documentation with @doc atm, since we only have to delete couple of words to match it with the new notation.

[TeroFrondelius]

Here is our documentation hosting / building system: https://readthedocs.org/
/CC @ovainola @ahojukka5

[TeroFrondelius]

And here is the official package for https://readthedocs.org//: https://github.com/JuliaLang/JuliaDoc

[ovainola]

Latest documentation page generated in readTheDocs: http://juliafem.readthedocs.org/en/latest/

[ahojukka5]

Ok we're almost done with this issue. We also need something to convert notebooks to rst files in order to progress. After little googling I found a workaround that might work

readthedocs/readthedocs.org#579

[quote]
arsenovic commented on Dec 20, 2013
Here is the Makefile directive i use to run and convert ipython notebooks to reST (followed by some minor sedding).

notebooks:
for file in ./source/tutorials/.ipynb; do
runipy -q --pylab -o $$file;
done
cd ./source/tutorials;ipython nbconvert --to rst *.ipynb
sed -i 's/``/`/g' ./source/tutorials/.rst
I suspect that a lot of projects will start using ipython notebook-based docs, because of their ease.

a conversation about this can be found here
http://python.6.x6.nabble.com/IPython-User-usage-of-notebooks-in-sphinx-td5041731.html
[\quote]

[TeroFrondelius]

@ahojukka5 check this: http://juliafem.readthedocs.org/en/latest/getfem_examples/README/

I would claim it's almost working system already. (Obviously some mistake in configuration, because it's called REAME. )

[ahojukka5]

Well it's making reference back to GitHub, not rendering the notebooks to html. At least for functional testing the notebooks should be run and checked that they compile.

I was thinking something like this, where notebooks are properly rendered to html:

http://www.damian.oquanta.info/posts/zen-themes-updated.html

Almost forgot, is it possible to get readthedocs url -> juliafem.org/docs?

[TeroFrondelius]

@ahojukka5 see #7

Here is good news for our own domain
http://read-the docs.readthedocs.org/en/latest/alternate_domains.html

[TeroFrondelius]

@ovainola why we don't see our documentation from https://github.com/JuliaFEM/JuliaFEM/blob/master/src/JuliaFEM.jl in our newly generated http://juliafem.readthedocs.org/en/latest/

[ovainola]

@TeroFrondelius since readTheDocs atm generates documentation from the /docs forder. This feature still needs some configuration.

[TeroFrondelius]

This is an alternative https://github.com/MichaelHatherly/Lexicon.jl. It contains doc test as well as some sort of scoring system.

[ovainola]

Looks cool. Magic in the documentation seems to be in the https://github.com/MichaelHatherly/Lexicon.jl/blob/master/docs/build.jl file, which runs all the tests and generates .md files into docs/api/ folder.

[ovainola]

I made some googling and found packages https://github.com/MichaelHatherly/Lexicon.jl & https://github.com/MichaelHatherly/Docile.jl and they seems to be closely linked: Docile provides the @doc macro and Lexicon build and generates the standalone documentation. These packages have more commits & issues (about 10 open vs 60 closed) compared to JuliaDoc ( 2 open and 6 closed) so I'd say we should give them a try.

[ovainola]

Ok, I tried Lexicon.jl and it's quite easy to use. But there's a catch: It's quite heavily linked to Doctile and in the source file it uses @document macro. This forces all the strings above functions to act as docstrings so it's basically automatic @doc for all the string. Here's an example how docstrings have to be written with Lexicon: https://github.com/QuantEcon/QuantEcon.jl/blob/master/src/lqcontrol.jl.

It also comes with doctests which is nice.

I forked the repository where I've been testing and here's a output of Lexicon: http://juliafemjl.readthedocs.org/en/latest/

note: docstring are written with numpy style and lexicon create markdown which has different syntax.

[TeroFrondelius]

@ovainola will the question mark "?" work in REPL with Lexicon.jl?

[ovainola]

Update: works

julia> using JuliaFEM

help?> JuliaFEM.elasticity_solver.calc_local_matrices!
calc_local_matrices! (generic function with 1 method)

julia> using Lexicon

help?> JuliaFEM.elasticity_solver.calc_local_matrices!
calc_local_matrices! (generic function with 1 method)

[method]

JuliaFEM.elasticity_solver.calc_local_matrices!(X,  u,  R,  Kt,  N,  dNdξ,  λ_,  μ_,  ipoints,  iweights)

  Calculate local tangent stiffness matrix and residual force vector R = T - F

 Details:

    source: (17,"/home/olli/.julia/v0.4/JuliaFEM/src/elasticity_solver.jl")

and if you have already @doc in front of docstring, Lexicon won't create any documentation..

[ovainola]

I'd say we give it a try with Lexicon & markdown syntax. If all in favor, I'll make a change in contributing.md

[MichaelHatherly]

Hi all, hope you don't mind my input here. When new packages are registered in METADATA I generally try to have a look to see what their plans are for docstrings.

It contains doc test as well as some sort of scoring system.

The doctest function is pretty basic currently, and just runs each code block in each docstring and records whether they "passed", "failed", or were "skipped". "Skipping" happens when the language isn't set to julia for a code block.

they seems to be closely linked

Yes, Docile provides the backend collection of docstrings while Lexicon handles the presentation.

It's quite heavily linked to Doctile and in the source file it uses @document macro.

Recently the need to use @document was removed. Docile now documents any imported package that contains bare docstrings without needing to be imported into that package.

will the question mark "?" work in REPL with Lexicon.jl?

If you'd like that to work for your users immediately then I'd recommend just adding import Lexicon to your toplevel module. It'll add a slight increase to your package load time, but given you're writing a FEM package it probably won't account for the majority of the total time.

and if you have already @doc in front of docstring, Lexicon won't create any documentation..

Docile will document a package that contains only @doc or only bare docstrings, not both. Given that JuliaLang/julia#11836 was merged a few days ago I'd recommend just using bare docstrings, which should work in both 0.4 and 0.3 (with Docile's help).

If you run into any problems feel free to ping me with @MichaelHatherly.

[TeroFrondelius]

@MichaelHatherly thanks for your comments. Do you know any good example repository, where Lexicon is used together with readthedocs? @ovainola has been studying more this topic, but I will try to briefly describe the wanted process:

1. New commit tricks the readthedocs
2. readthedocs virtual machine clone JuliaFEM repository
3. readthedocs virtual machine runs script to generate API documentations
4. readthedocs virtual machine runs doctests
5. readthedocs virtual machine converts Jupyter notebooks to html
6. readthedocs virtual machine runs our custom script to collect information etc. from README files
7. readthedocs virtual machine runs spinhx

[MichaelHatherly]

Basically, Lexicon just extracts the docstrings from your package and dumps them into markdown files.

What you do from there is completely up to you. I'm not sure whether you'll be able to get the readthedocs server to do (3) or (4) since that would require having julia present on their end. For (5) - (7): possibly, I've never tried doing that.

What I've been doing is just using mkdocs, which readthedocs supports, to do the presentation of the generated markdown files. I know Sims.jl has a build.jl script that does some more complex things. There's also some other packages listed on in Docile Readme some of which may help.

---

In your REQUIRE file I see you're only supporting Julia 0.4 and up, in which case there's quite a bit of work we're doing on updating the Julia manual and standard library docs right now, which should also be suitable for generating package documentation. So you might what to keep an eye on developments there.

[ahojukka5]

I haven't followed this conversation very closely but here's my contribution for subject.

In another project I did automatic API documentation generation using Sphinx and Buildbot. In principle it was doing make sphinx-apidoc && make html in makefile which was automatically executed when buildbot polled source and noticed changes. It was one side communication, i.e. buildbot didn't have to pull anything to repository but api documentation was generated on the fly before building documentation. I think we are planning to do something similar with this project. It seems that using readthedocs differs a bit form this approach?

[MichaelHatherly]

The readthedocs docs explain it a lot better than I'd be able to:

https://read-the-docs.readthedocs.org/en/latest/builds.html

and

https://read-the-docs.readthedocs.org/en/latest/webhooks.html

are probably the most relevant links for what you're interested in doing.

[ahojukka5]

Ok I almost got working automatic documentation generation (including notebooks!) at juliafem.kapsi.fi. With @ovainola we found several issues related to simple decision whether to use markdown syntax or restructured text format. rst + ipython nbconvert + sphinx is one possibility. nbconvert markdown support is somehow broken, notebooks are not rendered properly. mkdocs uses markdown syntax. Github supports both markdown and rst. So if we get automatic api generation working with rst then I don't see any problems. Using markdown the biggest problems is how to render notebooks.

[ovainola]

@MichaelHatherly Does Lexicon care what kind of syntax you're using inside docstrings? Atm we're using Markdown syntax but If we'll change to reStructuredText does it have some kind of effect? Of course we'll have to change the file suffix to .rst...

[MichaelHatherly]

Lexicon uses the Markdown.jl package for docstring parsing, so you'll need to use markdown syntax. There is support for changing the format of docstrings but until there are packages that can parse other formats you'll need to stick with markdown. As far as I know there's no rst parser in the METADATA yet.

[ahojukka5]

Ok that seems to be issue. Sphinx is not able to parse markdown.

1. convert markdown to rst before build using pandoc or similar?
2. find/develop rst parser for julia?

If we end up to write some very primitive parser by ourself (it's doesn't need to be very advanced if we don't use anything special in our docstrings), how do we use it with Lexicon?

[MichaelHatherly]

Sphinx is not able to parse markdown.

I believe there is some support for it quite recently: sphinx-doc/sphinx#1747

find/develop rst parser for julia?

That would be really cool to have. The most important part of Markdown.jl is that it renders markdown docstrings rather nicely in the REPL.

how do we use it with Lexicon?

You'd need to add a .docile config file to you're src/ directory with something like this in it:

import Docile.Interface: Docs, data, parsedocs
import ReST # ?

parsedocs(docs::Docs{:rst}) = ReST.parse(data(docs))

Dict(
    :format => :rst,
)

where ReST.parse converts a raw docstring to some kind of intermediate representation that has the necessary writemime methods defined for it. Placing the Dict last in the file is important.

[ahojukka5]

@MichaelHatherly Could you point me the relevant part of code where this is done using Markdown? As far as I understood this ReST.parse takes docstring as input argument and returns some "object" which have method writemime which again returns something when called. I could give a try for this approach because if we use numpy docstyle I think there's not so much to parse, it's already almost rst. But for that in need more information about this writemime.

[MichaelHatherly]

Lexicon hooks into Docile's parsing here. That's the equivalent to the line

parsedocs(docs::Docs{:rst}) = ReST.parse(data(docs))

that you'd put into a .docile file.

As far as I understood this ReST.parse takes docstring as input argument and returns some "object"

Markdown.parse is defined here and takes an AbstractString and converts to a Markdown.MD object defined here. There's several layers of calls that you'll need to walk through to get to the actual parsing mechanisms.

which have method writemime which again returns something when called

All the rendering methods are here. writemime methods are defined to write to an IO object which could be STDOUT, a file, or perhaps an IOBuffer. They don't actually return anything significant.

(Note that I've linked to the copy of Markdown.jl found in Lexicon. The package has been moved into Julia Base as of 0.4.)

[ahojukka5]

If we want to use numpy docstyle in our code it sounds a good idea to use numpydoc package with sphinx to get things working. Maybe @pv could give us some ideas how to combine julia docstrings with numpydoc..?

[pv]

You can probably reuse at least parser and formatter. You may need to modify a few parts IIRC there are autodoc style assumptions on working on Python importable objects. sphinx.ext.napoleon also has a separate implementation of Numpy style parser, don't know if it's tied to Python autodoc or not.

[ahojukka5]

I tried the following with numpydoc and sphinx.ext.napoleon (the latter also supports numpy documentation style)

Foobar
------

.. py:function:: src.codes.make_sum(a, b)

    Sum two numbers.

    Parameters
    ----------
    a : float
        First number.
    b : float
        Second number.

    Returns
    -------
    float
        sum of numbers `a` and `b`.

Using numpydoc this rendered like it should be while using sphinx.ext.napoleon it gives SEVERE: Unexpected section title. errors (napoleon functionality is linked to autodoc what we cannot use with julia).

So what we need to do to get this working is that by using Lexicon save/update we create function header info .. py:function:: src.codes.make_sum(a, b) (actually jl:function here..?) and write the rest of the docstring as it is to file. Doesn't sound too cumbersome.

Then we add numpydoc to conf.py and if everything goes like in movies we should get our api documentation automatically generated.

[ahojukka5]

Almost working

Docile: caching 5 modules from 'JuliaFEM'.
INFO: writing documentation to api/JuliaFEM.rst
INFO: writing documentation to api/JuliaFEM.abaqus_reader.rst
INFO: writing documentation to api/JuliaFEM.elasticity_solver.rst
INFO: writing documentation to api/JuliaFEM.interfaces.rst
INFO: writing documentation to api/JuliaFEM.xdmf.rst
ERROR: LoadError: MethodError: `save` has no method matching save(::ASCIIString, ::Base.Multimedia.MIME{sy                              mbol("text/rst")}, ::Lexicon.Index, ::Lexicon.Config)
Closest candidates are:
  save(::AbstractString, ::Base.Multimedia.MIME{symbol("text/rst")}, ::Lexicon.Index, !Matched::Lexicon.rs                              tConfig)
  save(::AbstractString, !Matched::Base.Multimedia.MIME{symbol("text/html")}, ::Lexicon.Index, ::Lexicon.C                              onfig)
  save(::AbstractString, !Matched::Base.Multimedia.MIME{symbol("text/md")}, ::Lexicon.Index, ::Lexicon.Con                              fig)
  ...
 in save at /home/users/thdr/.julia/v0.4/Lexicon/src/render.jl:237
 in anonymous at /home/users/thdr/sites/juliafem.org/buildbot-slave/runtests/build/docs/build_api.jl:49
 in cd at ./file.jl:22
 in include at ./boot.jl:254
 in include_from_node1 at loading.jl:133
 in process_options at ./client.jl:306
 in _start at ./client.jl:406
while loading /home/users/thdr/sites/juliafem.org/buildbot-slave/runtests/build/docs/build_api.jl, in expr                              ession starting on line 43
make: *** [apidoc] Error 1

[ovainola]

should work now. Problem was the save function with index.rst, which I had commented out in my local repo. I made a commit where I commented the line. Problem isn't fixed yet, but build should pass now

[TeroFrondelius]

Possible interestin package https://github.com/jakebolewski/JuliaParser.jl

[ovainola]

Finally, automatic documentation works! Docstrings are read using Lexicon & generated into html with sphinx. Url to documentation: http://www.juliafem.org/api/JuliaFEM.elasticity_solver.html

[ahojukka5]

This was our last issue in "getting started".