Styled markdown, with a few tweaks

[tecosaur]

With StyledStrings and JuliaSyntaxHighlighting we can make the rendering of Markdown.MD:

• prettier
• customisable
• more composable

Since this is mostly about rendering, here are some screenshots:

Current

New default

New customised

---

Depends on #51807, #51810, and #51869

[tecosaur]

What do you know, all the dependent PRs are now merged 🥳. We can now think about reviewing and merging this.

[tecosaur]

I'll push a version with the fixed pkgimage.mk and tweaks based on API changes that happened in #51807 once #53155 is merged.

[tecosaur]

@vtjnash any further thoughts on the current version?

[vtjnash]

This seems a bit suspect to imply that applying a face! to the result of eachmatch will alter the string hl

[tecosaur]

When taking a substring of an AnnotatedString, the underlying string is not mutable, but the annotations are. This is rather convenient when you want to search for patterns to style (no other convenient patterns are currently possible, actually).

[vtjnash]

There does not appear to be a field ncodeunits (it is a function)

[tecosaur]

The match is of type SubString{AnnotatedString}, so it indeed has a field ncodeunits. May as well use the function though.

[vtjnash]

It feels a bit tricky to have a full REPL here, but anyways, I think this is supposed to use Base.parse_input_line aka Meta.parseall here and repeatedly add lines until the expression is complete. The result may not be quite the same as eachmatch would return, but may be closer to what eachline returns, followed by if ismatch followed by Meta.parseall consuming more lines until it reaches a concluding statement (a blank line or a complete expression or a parse error)

[tecosaur]

This is modelled after the current behaviour of OhMyREPL which seems to work. I think Kristoffer and I had a brief conversation on this on Slack where I asked "why Meta.parse not Meta.parseatom?" but I forget the details and it's been swallowed by the Slack-hole at this point 🥲.

I'll also note that it's actually somewhat difficult to tell the behavioural difference between parse, parseatom, and parseall as the latter two have no docstrings and just call Core._parse, which itself seems to have no (useful) comments.

[vtjnash]

Isn't there an optimized version of this, for copying between IO objects without the intermediate copy?

Suggested change

	print(io, read(seekstart(a.io), AnnotatedString))
	write(io, seekstart(a.io))

Or is the problem that io might be wrapped up somewhat, so that dispatch currently won't reliably know if it is annotation-capable and end up discarding those?

[tecosaur]

io may be stdout, and we currently don't have a specialised print/write method for AnnotatedIOBuffer to .

I'd suggest that we perhaps add a "TODO change to direct write after implementing specialised AnnotatedIOBuffer printing"-type mental note or comment so we can come back to this if/when we do so.

[vtjnash]

Actually, could you drop this specific particular part of the PR, merge everything else, and then add this in a later PR? It is a bit of a bonus feature, and I don't want it to hold up everything else

(perhaps just add it to the list along with "julia" to get some primitive highlighting, or just leave it bare for now)

[tecosaur]

Sure, I'll bundle it with julia for now.

[tecosaur]

For future reference, this is the code being removed:

    elseif md.language == "julia-repl" || Base.startswith(md.language, "jldoctest")
        hl = AnnotatedString(md.code)
        for (; match) in eachmatch(r"(?:^|\n)julia>", hl)
            StyledStrings.face!(match, :repl_prompt_julia)
            afterprompt = match.offset + ncodeunits(match) + 1
            _, exprend = Meta.parse(md.code, afterprompt, raise = false)
            highlight!(hl[afterprompt:prevind(md.code, exprend)])
            if (nextspace = findnext(' ', md.code, exprend)) |> !isnothing
                nextword = hl[exprend:prevind(hl, nextspace)]
                if nextword == "ERROR:"
                    StyledStrings.face!(nextword, :error)
                end
            end
        end
        hl

[vtjnash]

Suggested change

	code = if md.language ∈ ("", "julia", "julia-repl", "jldoctest")
	code = if md.language ∈ ("julia", "julia-repl", "jldoctest")

just arbitrary <raw> blocks marked with ``` shouldn't get julia-formatted, right?

[tecosaur]

I guess it depends how you interpret them. I tend to see them as "whatever the default language is" which on GitHub is text and in Julia docstrings is julia.

Ultimately, it's a heuristic choice, and I may well have gone the wrong way here.

[vtjnash]

I think raw is the conservative choice here to stick with

[tecosaur]

Sure. It would be nice to syntax highlight function signatures though, which usually appear like so:

"""
    annotate!(char::AnnotatedChar, label::Symbol => value)

Annotate `char` with the pair `label => value`.
"""

[vtjnash]

True. Probably should be a separate PR though, since it is potentially a more significant change that downstream consumers would not be able to distinguish between annotated and un-annotated content

[tecosaur]

Yes, the initial code element of the docstring would be highlighted as Julia code, if that's what you mean.

[tecosaur]

@KristofferC might have some helpful comments here, since OhMyREPL does Julia syntax highlighting of markdown code blocks.

[vtjnash]

It seems like we would need a way to distinguish arbitrary Markdown that is being formatted from specifically Julia docstrings that are being rendered

[tecosaur]

The only way to conclusively tell, is to use language annotations on code blocks. Short of that, it's a guess/default. I think "guess Julia because we're in Julia" and requiring raw/text to be explicit makes sense, but as you demonstrated earlier that's not the only reasonable position.

[KristofferC]

If defaulting to rendering Julia syntax gives us highlighting in the docstrings and otherwise not, then I think we should default to Julia.

[fingolfin]

@vtjnash you added a "merge me" label but did not approve the PR... Should this be read as an "implicit approval"?

[fingolfin]

There seem to be genuine errors in the CI job logs

[tecosaur]

The Pkg CI errors will be resolved once JuliaLang/Pkg.jl#3884 is merged, and Pkg bumped.

[tecosaur]

We might be in the clear now! 🎉 🤞

[tecosaur]

If there are no objections, I'll merge this shortly.

[mortenpi]

Maybe just a quick check to do if you have the branch compiled: do MarkdownAST and Documenter test suites pass with this?

[tecosaur]

Thanks for that suggestion Morten, I do indeed and the results are:

• MarkdownAST: all tests pass
• Documenter: fails to load due to libgit2 error

[mortenpi]

Documenter: fails to load due to libgit2 error

Interesting, but that seem completely unrelated, and so I think we can ignore that. Documenter's reliance on the Markdown stdlib should mostly be covered by the MarkdownAST tests anyway. Thanks for checking!

[tecosaur]

Since it's been a while, Triage thinks we should just let this appear in 1.12.