move Markdown to stdlib

[JeffBezanson]

I'm putting this up at a pretty early stage so folks know I've decided to do something this crazy, before anyone else attempts such a foolish errand.

I have this to a point where I can build, start the repl, and successfully do ?sin. Safe to assume more work is needed. I might need help at some point.

A lot of the code in Base at the intersection of the docsystem and markdown has to do with searching and displaying documentation, so for now I've moved that code to the REPL package under docview.jl. We might want to do more refactoring in the future.

[JeffBezanson]

One question is what to do with the apropos function. Fortunately, all of the other doc display features are triggered by ? and so are REPL-specific, but apropos is a function exported from Base. Given the search: and "Perhaps you meant" features of REPL help, could we just discontinue it?

[JeffBezanson]

A couple other questions that this raises:

• Do we still need the ability to use -> syntax for doc strings, or can that be removed?
• One thing that's really confusing in working on this code is that @doc is used for both defining and accessing doc strings. I'd like to change that.
• Is getdoc ever used? There don't seem to be any uses in base or stdlib.
• What is the doc_str macro (doc" ... ") for? Is it used anywhere?

cc @MichaelHatherly

[iblislin]

What is the doc_str macro (doc" ... ") for? Is it used anywhere?

It returns a MD object. Useful to prevent something wrongly escaped when writing latex in docstring.
e.g.

julia> """
       ```math
       \alpha = 1
       ```
       """
       g(x) = 42

julia> doc"""
       ```math
       \alpha = 1
       ```
       """
       h(x) = 42


help?> g 

lpha = 1

help?> h

\alpha = 1

[JeffBezanson]

Would a raw" " string work for that?

[iblislin]

Yeah, raw" " works for my example. But MD object has other features like treating multiple blank lines as only one line break when rendering.

[JeffBezanson]

That should be fine; we do markdown parsing lazily anyway, so I don't think it's necessary to create an MD object at the point where the doc string is written.

[JeffBezanson]

Looking closer, I see that the purpose of doc" " is to create a markdown object with interpolated julia values. The only difference between that and md" " is that doc" " adds location metadata to the markdown object. But I imagine that's not necessary, since docstrings work perfectly well without using doc" ". So I suggest deprecating doc" ", and requiring using Markdown if people want to write md" " doc strings.

[iblislin]

So I suggest deprecating doc" ", and requiring using Markdown if people want to write md" " doc strings.

I greped Documenter. It is using these metadata.
I'm ok with using Markdown, but I still want different macro for creating a general purpose MD object (which is clean and without metadata) and docstring MD object.

[JeffBezanson]

I greped Documenter. It is using these metadata.

But most doc strings (in Base at least) are written with plain strings, not doc" ". So are they missing needed metadata?

We can keep @doc_str for now, I'll just move it to the Markdown package. But I think it would be better to add the documentation metadata when the object is used as a doc string, instead of when creating it.

[martinholters]

Reading the above, I was under the impression that I should be able to replace doc""" ... """ with raw""" ... """ or md""" ... """ (modulo potential problems when using Documenter), but:

julia> using Markdown

julia> begin
           doc"""
           $\sqrt{a^2+b^2}=\pm c$
           """
           function f1 end

           raw"""
           $\sqrt{a^2+b^2}=\pm c$
           """
           function f2 end

           md"""
           $\sqrt{a^2+b^2}=\pm c$
           """
           function f3 end
       end
f3 (generic function with 0 methods)

help?> f1
search: f1 Inf16 ComplexF16 fld1 Float16 fldmod1

\sqrt{a^2+b^2}=\pm c


help?> f2
search: Float32 f2 Inf32 ComplexF32

  No documentation found.

  f2 is a Function.

  # 0 methods for generic function "f2":


help?> f3
search: f3 Inf32 ComplexF32 Float32

  No documentation found.

  f3 is a Function.

  # 0 methods for generic function "f3":

Am I being stupid here?

[iblislin]

I think you are right.

Although I'm not good at reading lisp, I guess it's here.

julia/src/julia-parser.scm

Lines 2354 to 2373 in 6c59599

	(define (doc-string-literal? e)
	(or (simple-string-literal? e)
	(and (pair? e) (eq? 'string (car e))) ; string interpolation
	(and (length= e 3) (eq? (car e) 'macrocall)
	(simple-string-literal? (caddr e))
	(eq? (cadr e) '@doc_str))
	(and (length= e 4) (eq? (car e) 'macrocall)
	(simple-string-literal? (cadddr e))
	(eq? (cadr e) '@doc_str))))
	(define (parse-docstring s production)
	(let ((startloc (line-number-node s)) ; be sure to use the line number from the head of the docstring
	(ex (production s)))
	(if (and (doc-string-literal? ex)
	(let loop ((t (peek-token s)))
	(cond ((closing-token? t) #f)
	((newline? t) (take-token s) (loop (peek-token s)))
	(else #t))))
	`(macrocall (core @doc) ,startloc ,ex ,(production s))
	ex)))

Seem @doc_str is special and deeply coupled.

[martinholters]

Um... ok... somehow, putting a macro that receives special treatment by the parser into a (stdlib) package seems wrong to me.

[StefanKarpinski]

Agree, although I think the main issue is the macro getting special treatment by the parser in the first place. Is the special treatment here to cause doc"..." strings to attach to the following object like plain string literals do?

[JeffBezanson]

Is the special treatment here to cause doc"..." strings to attach to the following object like plain string literals do?

Yes. I think it would make more sense just to allow any string or string macro though. Yes, that would allow silly things like r" ... " but not much harm there I'd say --- the doc system could still give an error for that.