Fix #11798 & load bootstrapped docs.

[MichaelHatherly]

I've yet to see how well this works within Base and whether I've missed any cases, hence the use of [skip ci] for now.

Approach taken:

Extract the signature from a method definition and use which to determine the Method that has been defined. This avoids referencing the Function prior to defining it, which was the cause of #11798.

[MichaelHatherly]

@one-more-minute this works for the original case in #11798 and a selection of random methods from base. If this is indeed a suitable fix then I'll add some tests and let it run on travis.

[MikeInnes]

This seems like a reasonable approach, but as an alternative I'm wondering whether there's a way to check if the binding has been reified yet – i.e., check whether the read definition will be treated as a new function, and if so, treat it as non-existent. Anyone know of a way to do that?

We'll probably need something like this anyway, it's a bit fiddly and there are a lot of edge cases to go over.

[MichaelHatherly]

wondering whether there's a way to check if the binding has been reified yet

Yeah, that would be a lot cleaner. I'll resolve the conflicts here in any case.

[yuyichao]

Does this work for parametrized functions and would it be possible to extend it (doesn't have to be in this PR) so that it's possible to document a method after it's definition? Either by documenting on the signature of a method as a Expr or on a Method object.

I've also added document about only possible to document the whole function after it's defined so you might want to change that if necessary.

[MichaelHatherly]

Does this work for parametrized functions

It should do, using a let block and manually created TypeVars. It's worked for everything I've tried so far... ;)

so that it's possible to document a method after it's definition

Syntax I used in Docile for that was

"..."
(f, Foo, Bar)

to document f(::Foo, ::Bar). Not the nicest syntax though. Perhaps following your quoted macros would be best:

"..."
:(f(::Foo, ::Bar))

That should be a relatively simple addition to this PR if we do happen to go this route.

[yuyichao]

"..."
:(f(::Foo, ::Bar))

Yeah, that's more or less what I'm thinking about and also documenting the Method object. Not sure which one would be more convenient although I don't see why we can't support both... =)

Use case here #12000 (comment)

[yuyichao]

And the reason I asked about parametrized method is that I'm wondering if you could hit some bug for comparison involving bound TypeVar

[MikeInnes]

@MichaelHatherly We can't use the current trick for docs created during bootstrap anyway, so if you get this working solidly I'm happy to switch (even if it's a little less pretty).

[MichaelHatherly]

documenting the Method object

and

Use case here #12000 (comment)

Directly in the module's ObjectIdDict? If so then the help system won't be able to retrieve the docs. We're storing Function and Method objects in FuncDocs keyed by Function:

julia> f(x) = x
f (generic function with 1 method)

julia> m = f.env.defs
f(x) at none:1

julia> "..."
       m
f(x) at none:1

julia> Docs.meta(Main)
ObjectIdDict with 2 entries:
  f(x) at none:1           => Base.Markdown.MD(Any[Base.Markdown.Paragraph(Any["..…
  #= circular reference =# => Base.Markdown.MD(Any[Base.Markdown.Paragraph(Any["Do…

help?> f # won't pick up the docs.
search: f fd for fma fld fft full find filt fill fft! fdio frexp foldr foldl flush

f (generic function with 1 method)

help?> m # will pick them up but not a suitable solution I think.
search: m mv mod min max map myid modf mod1 mmap mean mark map! Mmap macro mtime

  ...

help?> f(::Any) # perhaps we can support this syntax?
ERROR: syntax: invalid "::" syntax

So there's no easy way to view docs for a documented Method that's not in a FuncDoc. That final syntax f(::Any) might be suitable for finding docs for a specific Method.

---

And the reason I asked about parametrized method is that I'm wondering if you could hit some bug for comparison involving bound TypeVar

I've not run into that yet, thanks for the heads up. Regarding not constructing TypeVars directly: is my usage here alright then, or should another approach be found?

[MichaelHatherly]

We can't use the current trick for docs created during bootstrap anyway, so if you get this working solidly I'm happy to switch (even if it's a little less pretty).

Ah... good point. I think this should be workable with a bit more testing (and an "ok" regarding constructing the TypeVars directly).

[MichaelHatherly]

That final syntax f(::Any) might be suitable for finding docs for a specific Method.

Just to clarify that comment: I'm aware that help?> f(1) does work fine since it calls @which, but in some cases constructing a value of particular type might be more difficult that just writing that type itself.

[MikeInnes]

For documenting methods after they're defined, would it be OK to just have –

"..."
f(::Foo, ::Bar)

? We could also support

"..."
@which f(1)

but honestly, I think that's a bit confusing.

@MichaelHatherly Is there any chance you'd be able to look at applying this work to the docstrings stored at bootstrap? Pre-bootstrap we only need to support a very limited subset of what the full doc system supports, e.g. just functions and methods, so it'd be great to push that forward here.

[MichaelHatherly]

but honestly, I think that's a bit confusing.

Yeah, I guess until someone actually needs it (which would probably be quite unlikely outside of Base), there's no point in adding even more complexity to the docstring rules.

Is there any chance you'd be able to look at applying this work to the docstrings stored at bootstrap?

Presumably all that needs to be done is something like:

for (mod, doc, def) in Base.DocBootstrap.docs
    eval(mod, :(Base.@doc($(doc), $(def))))
end

Doing that a the REPL gave me the following error:

ERROR: could not open file /home/mike/interp.jl
 in include at ./boot.jl:254
 in include_from_node1 at ./loading.jl:200
 in anonymous at no file:2

which is from

julia/base/markdown/Julia/Julia.jl

Lines 3 to 10 in e66175b

	"""
	This file contains markdown extensions designed to make documenting
	Julia easy peasy.
	We start by borrowing GitHub's `fencedcode` extension – more to follow.
	"""
	include("interp.jl")

. I guess we need some additional logic to filter out those...

Other than that error the rest of the docstrings that are currently written inline look like they're in __META__ now. Where in the build process should this be done, presumably as late as possible?

[StefanKarpinski]

We could also support

"..."
@which f(1)

but honestly, I think that's a bit confusing.

It would be significantly less confusing if @which were renamed to @method.

[MichaelHatherly]

Commit two adds a first try at loading bootstrapped docs stored in DocBootstrap.docs into their correct __META__ locations.

At the moment the code is a little repetitive and I'll try to simplify it if this approach is suitable.

[MikeInnes]

Ok, I like the define approach. What would make it a lot simpler is if in docm we can just do define || (def = nothing), then just pass def to each of the sub functions as before. Looks like funcdoc is currently dependent on def′ for picking up the signature – in that case we can just pass in the sig as a third argument, and that dovetails nicely with implementing method syntax for helpdb.jl.

Loading bootstrap docs can happen as soon as the doc system is loaded, so it might be better to have it just after the @doc macro is replaced.

[MichaelHatherly]

What would make it a lot simpler is if in docm we can just do define || (def = nothing), then just pass def to each of the sub functions as before.

Done.

[MichaelHatherly]

This is good to go as far as I can tell. Fixes #11798 as well as #12437, which is one of the issues listed as blocking the 0.4 RC mentioned in #11536.

[MichaelHatherly]

Not sure what's up with appveyor, rebased and pushed again to hopefully fix that.

[MikeInnes]

Fantastic, this will really streamline the process of moving docs into Base.

[MichaelHatherly]

Great, thanks. @one-more-minute, do we also want the syntax for adding docs to methods after definition similar to the way macros can be?

As in:

f(x::Int, y) = ...

"..."
:(f(::Int, ::Any))

I've got some working code for that somewhere.

[MikeInnes]

That would be great, but is there a reason it needs to be quoted? I think it's ok to disallow documenting the result of a function call.

[MichaelHatherly]

I'm fine with having it unquoted, was just following the macro version.

[yuyichao]

I'm thinking the quote version can be easily wrapped in a macro (i.e. a custom macro that define a method can just evaluate to the signature expression of the function) and the @doc macro can juat parse the return value of the expression to be documented.

i.e. would be nice if something like this can work

macro def_f()
    quote
        $(esc(:(f(a::B, c::D, e::F...) = 1)))
        # many random code
        :(f(a::B, c::D, e::F...))
    end
end

"""
some doc for `f`
"""
@def_f

[MikeInnes]

Ok, so you're imagining that if someone tries to document the value :(f(...)) it ends up documenting the specified method instead of the Expr object itself. It could work, but it's technically separate from the syntax dispatch in docm so I think we can discuss it separately.

[yuyichao]

Right. Another possibility is to allow documenting the Method object.

it's technically separate from the syntax dispatch

Agree. I just imagine that most of the time people need to document a method after its definition is because the function is defined in a macro. Would be nice if the macro auther can handle this such that it can be documented just as a normal function definition.

[MikeInnes]

In general we're moving away from allowing values to be documented via @doc at all – it makes sense to just be explicit and use the doc! function for values, and it's not really less convenient.

I also think that documenting a value should always just do that. If you special case certain kinds of expressions, for example, then

doc!(:(2+2), md"foo")

and

doc!(:(f()), md"foo")

have totally different semantics, which strikes me as a recipe for confusion.

I see your point and it would be good to have a solution to that problem, though.

[yuyichao]

Personally I don't need this so I don't have much detail about the general use case. Ping @mauro3 since he is the first who brought this up in #12000 (comment)

[MichaelHatherly]

Since @doc just tries to document the resulting value of a block if it doesn't know what to do, we can define additional doc! methods to handle macros on a case-by-case basis:

julia> type MacroDocs
           func
       end

julia> macro makething(f)
           quote
               $(esc(f))(a) = 1
               $(esc(f))(a, b) = 2
               MacroDocs($(esc(f)))
           end
       end

julia> function Base.Docs.doc!(md::MacroDocs, meta)
           f = md.func
           Docs.doc!(f, meta)
       end
doc! (generic function with 6 methods)

julia> "..."
       @makething(f)
  ...

help?> f
search: f fd for fma fld fft full find filt fill fft! f...

  ...

julia> Main.__META__[f]
Base.Docs.FuncDoc(...

A very contrived example, but the general idea is for each macro the author would define an object, here MacroDocs, to return any necessary data. They also define a new doc!(::MacroDocs, ::Any) method that then handles what to do with that data. In this case just creating the usual FuncDoc object.

[yuyichao]

I love this idea!
So basically clean up and document some internal functions and we should be all set.

I didn't check how exactly would you construct FuncDoc, might be a good idea to make is easier to be used in this case. (Or not worry about it if it is hard to cover most of the use case.)

[MichaelHatherly]

You can put anything as the value, so it doesn't have to be a FuncDoc. Could be some custom type with display methods defined on it. In that example calling doc!(::Function, ::Any) builds a FuncDoc.

If this way works for those who need it, then clean up and documenting it would be good.

[yuyichao]

You can put anything as the value, so it doesn't have to be a FuncDoc.

I mean a helper would be nice so that people don't need to write their own one. (Although that's certainly not fundamental).

[MikeInnes]

Since @doc just tries to document the resulting value of a block if it doesn't know what to do

It's a nice idea, but I'm not sure we should keep that fallback just for this feature. I'm concerned that as we add more syntax to @doc, it just gets harder to predict what will end up being documented as a value and what won't. I'm leaning towards having a well-defined set of allowed syntaxes and erroring on anything else, which has the added benefit that adding additional syntax isn't a breaking change.

It might be a lot simpler to just pass the docstring to the macro that defines the function.

Here's another option:

macro foo()
  def = :(function f() end)
  quote
    $(Expr(:meta, :doc, def))
    $def
    # something fancy
  end
end

"foo"
@foo

docm then scans for the doc meta and, if found, documents that expression alone, while still running the entire expression.

[MichaelHatherly]

Here's another option:

That seems reasonable as well. So long as whatever we choose is simple for people to hook into. From your example I'd say it's less of a hassle than having to define additional types just be able to hook into the docsystem and would probably provide better error messages.