Touching Base

[MikeInnes]

Ok, I think we're about ready to push forward with getting this stuff in Base. A rough plan of action:

1. @MichaelHatherly You send a PR to get the @doc macro in. This should be fairly simple; I think @stevengj is right that being completely generic is the right way to start out, and it actually makes the whole system simpler if you don't have to worry about display at all. So @doc just associates whatever object it's given with whatever function, help(f) retrieves it.
2. In parallel, I'll send a PR for integrating Markdown.jl, as well as implementing the mdoc"" macro which will store a smartly-displayed string.
3. Once Markdown is in you can make @doc convert plain strings to mdoc"" strings by default.
4. Then we can start to think about documenting methods, bugging Jeff about custom syntax, etc. But we'll have something basic working either way.

Sound like a plan? Really it looks like we're 99% of the way there by now, it's just a case of getting it into Base.

If you want I'm happy to send Markdown.jl + @doc as a single PR myself, too.

[sglyon]

This is very exiting. Good work guys!

[stevengj]

Can we please just use md"..."? Markdown strings aren't just for documentation, and md is shorter to type.

[stevengj]

There is also the question of whether @doc in Base should know about metadata. The consensus seems to be that it should not know about metadata, at least for now. If we store generic objects, however, something like my MetaDoc suggestion in JuliaLang/julia#8514 would allow storage of metadata without @doc knowing or caring.

[MichaelHatherly]

Yes, I'll get something together tomorrow if time allows. I'd definitely like to try PR this part myself -- learning experiences and all that jazz :)

@one-more-minute you mention associating docs with functions. What about methods, etc? I'm also in agreement with @stevengj that md"..." is a nicer choice of name.

I was also contemplating a few changes today, namely a @meta that would take the Dict{Symbol,Any} and associate that with the object. :doc would be used to store the docstring. (Am I correct in thinking Clojure does a similar thing?). @doc would then just be a special syntax for the more general @meta macro taking a "..." or """...""" and adding it to :doc. I haven't thought these through much though, sound remotely reasonable to you guys?

Also, I'd like to avoid the @docstrings call currently needed. Is there a way to add it to the default module initialisation? I've not delved into Base that much yet, not sure about this one.

[MichaelHatherly]

Wrapping the Dict in a MetaDoc/Metadata would probably be good.

[stevengj]

If you have MetaDoc(md"...", [:author => "foo", ...]) then you don't need @meta.

I think everyone agrees we need the option of separate Function (generic) and Method (specific) documentation.

(I'm confused about this issue... it seems to be somewhat redundant with issue 8514)

[MikeInnes]

@stevengj Markdown.jl already has an md"" macro for general use, but it has to parse eagerly to be efficient and interpolate eagerly in order to respect scoping rules. My understanding was that you wanted docstrings to behave differently, e.g. loading interpolated code lazily. And that's before we even get to things like adjusting image URLs to be correct relative to the source path. We need a separate string macro to do these things.

And of course, I chose the documentation string macro to have the longer name because the @doc macro will mean that no one ever actually has to type it. Please, please tell me we're not going to bikeshed the name of something no one ever has to type.

@MichaelHatherly I definitely agree that one docstring per function is overly limited, and certainly have no objection to the various metadata proposals. But I think for now we just need to get the simplest thing that works into Base. Once that happens people can start using it and we can iterate as necessary.

Starting simple will probably also help with backporting to 0.3, which IMO would be really valuable.

[MikeInnes]

@MichaelHatherly Oh yeah, and about the @docstrings call – modules do a bit of setup already (e.g. setting up the eval function, importing Base operators etc.) so it should definitely be possible. I'm not sure where that's defined though, your best bet is to ping Jeff when you send in the PR I think.

OTOH you could also just use a single metadata dict in Base. Can I ask why you're using a module-specific one? It doesn't really matter that much as long as it works, anyway.

Edit: Oh yes, you want module documentation to be compiled with the module.

[MichaelHatherly]

If the consensus is building for adding just the basics to Base for the moment, then I believe that @mauro3's JuliaLang/julia#5572 PR should cover associating docstrings with objects (if I've read correctly). What's the current status on that PR? I'd rather not duplicated work that's already waiting to go into base.

[MikeInnes]

It looks pretty stale to me. I'd suggest just checking out that branch and seeing if it still works. If it does you can build on it, fix remaining issues and send in your own PR. But it may be too much effort to make it current, in which case you can just start from Docile.

Honestly, I'd scrap trying to integrate helpdb.jl / the Help module for now, aside from overloading help() to check helpdb and the meta dict. Migrating Base docs is one of the things we can look to doing in the second iteration, once the basic system is working.

[mauro3]

Movement, exciting! I fixed & rebased JuliaLang/julia#5572, take what you need. And let me know if I can help.

Note that the storage backend is not (yet) what the higher powers would like it to be: Over in JuliaLang/julia#3988 (comment) , @StefanKarpinski suggested an implementation of metadata storage system with a separate help/meta-dict for each module.

I will mull over what that proposal means in terms of the function-interface to the metadata and will do the changes (if there are any) over the next few days. Swapping out the backend will be more work and I can do that in the next week or two.

[MichaelHatherly]

Excellent, thanks @mauro3!

[stevengj]

The @doc macro does not and cannot obviate the need for typing md"...", because strings are parsed before passing them to macros. And since we need documentation strings that allow us to type $ and \ without substitution, we need to use explicit string macros (absent changes to the Julia parser).

md"..." in the current Markdown.jl may not do what we need (creation of a lazily parsed string with no interpolation), but that doesn't mean it needs to be merged into Base in this form.

[stevengj]

Ah, you're quite right; I had thought that all the string parsing happened beforehand, but I was mistaken.

[ivarne]

No, @stevengj was right. @show does pretty printing of the string concatenation. Try the same with dump() to get the actual AST, where you can see that the string is parsed, and contain the LineFeed character '\n' == '\xA' from parsing \n.

julia> macro test (ex); dump(ex); println(ex.args[3]); ex end

julia> @test("a $o 4\nd")
Expr 
  head: Symbol string
  args: Array(Any,(3,))
    1: ASCIIString "a "
    2: Symbol o
    3: ASCIIString " 4\nd"
  typ: Any
 4
d
ERROR: o not defined

Anyway; the $o variable is not accessed and if @show can glue the string expression together again, so can @doc. The much bigger problem is how to see the difference between a literal \n in a docstring and a actual newline. For strings we can't do that, and require users to escape the \ and write \\n. I don't think that is desirable in a docstring, especially if someone were to use LaTeX for their equations.

[MikeInnes]

We could escape newlines by default in regular strings and even do things like only escaping it in multiline strings within LaTeX equations only.

Even so I think we're trying to solve a problem we don't have here. Firstly, because although LaTeX and interpolation aren't completely supported yet, neither are blocking the system from being useful in its current state. And secondly, while I'll be the first to agree that @doc "..." -> is way too cumbersome, Stefan and Jeff have both expressed support for having this in the language, which means we can just replace plain strings with @doc mdoc"..." -> eventually, solving all our issues at once.

[stevengj]

As @ivarne already wrote in #8514, having two kinds of string literals — "..." ignores escaping in a documentation context but not elsewhere — would be confusingly inconsistent. So much cleaner to just type md.

I really think this issue should be closed; at this point it is just duplicating the discussion in #8514, and we shouldn't be having the same discussions in two places.

[MikeInnes]

Happy to do that.