Add TODO lists support for Blackfriday (*.md files)

[bep]

See #2022
See shurcooL/github_flavored_markdown@2b74a5b

[bep]

Note: this is the top priority as Blackfriday is the main engine in use -- but it would be nice if this syntax was supported for both mmark and md.

[dmitshur]

It looks like hugo already has an internal blackfriday.Renderer interface implementation in helpers/content_renderer.go.

So you can just implement a custom ListItem method there, just like I've done in shurcooL/github_flavored_markdown@2b74a5b.

[bep]

Yes, we're not introducing another extension just to support TODO lists. And since implementing this means either implementing it or copying @shurcooL s lines, both of which should produce very similar results, I guess a copy of the method with a reference to the copyright owner above the method would be ok?

[dmitshur]

No need to worry about giving me credit explicitly in the code, feel free to use my code in the most convenient way. It's trivial code anyway, just a switch statement with some cases.

[phtan]

I'll try to implement what @bep said . There are no takers for this issue so far, right?

[phtan]

So I tried to modify @shurcooL 's code but for some reason it is not adding the extra HTML.

Here's the result of a small test:

FAIL: TestRenderTaskList (0.00s)
    content_renderer_test.go:113: Actual rendered Markdown ("<li>- [x] done\n- [ ] to do</li>\n") did not match expected markdown ("(?s)^<ul>\\n<li><input type=\"checkbox\" checked=\"\" disabled=\"\"> done</li>\\n<li><input type=\"checkbox\" disabled=\"\"> to do</li>\\n</ul>\\n$") **

Here's my code in content_renderer.go:

// Task List support. 
 78 // Courtesy of Dmitri Shuralyov <shurcooL@gmail.com> who originally 
 79 // implemented this in his renderer 'github_flavored_markdown' 
 80 func (r *HugoHTMLRenderer) ListItem(out *bytes.Buffer, text []byte, flags int) { 
 81   switch { 
 82   case bytes.HasPrefix(text, []byte("[ ] ")): 
 83     text = append([]byte(`<input type="checkbox" disabled="">`), text[3:]...) 
 84   case bytes.HasPrefix(text, []byte("[x] ")) || bytes.HasPrefix(text, []byte("[X] ")): 
 85     text = append([]byte(`<input type="checkbox" checked="" disabled="">`), text[3:]...) 
 86   } 
 87   // jww.ERROR.Printf("Appended checkboxes. text: %s", text) 
 88   r.Renderer.ListItem(out, text, flags) 
 89 } 

[dmitshur]

Have you looked at the sanitization code? It's possible the current policy strips out the new tags that are needed for this.

If you look at my commit, it makes appropriate changes to the bluemonday sanitization policy.

@@ -52,6 +52,8 @@ func Markdown(text []byte) []byte {
    p.AllowAttrs("class", "name").Matching(bluemonday.SpaceSeparatedTokens).OnElements("a")
    p.AllowAttrs("rel").Matching(regexp.MustCompile(`^nofollow$`)).OnElements("a")
    p.AllowAttrs("aria-hidden").Matching(regexp.MustCompile(`^true$`)).OnElements("a")
+   p.AllowAttrs("type").Matching(regexp.MustCompile(`^checkbox$`)).OnElements("input")
+   p.AllowAttrs("checked", "disabled").Matching(regexp.MustCompile(`^$`)).OnElements("input")
    p.AllowDataURIImages()

    return p.SanitizeBytes(unsanitized)

Does hugo also use bluemonday?

If you uncomment jww.ERROR.Printf("Appended checkboxes. text: %s", text), does debug print statement get printed?

Also,

// Courtesy of Dmitri Shuralyov <shurcooL@gmail.com> who originally 
// implemented this in his renderer 'github_flavored_markdown'

I appreciate the nice gesture, but there's no need to pollute Go code files with such attribution comments. You have my full permission to use the (trivial) code without any comments about it. You've already given me attribution in this issue, etc.

[phtan]

Does hugo also use bluemonday?

No, it doesn't.

If you uncomment jww.ERROR.Printf("Appended checkboxes. text: %s", text), does debug print statement get printed?

Yes, the statement gets printed.

[phtan]

I'm wondering if there's a place where you list out the custom methods that you've written for the renderer. You know, to register the custom methods. Maybe that's what's missing

[phtan]

It's also strange that when I expect a <ul><li> ... </li></ul> to be produced, there is a <li> ... </li> instead

[bep]

@phtan I suggest you go to the Blackfriday site and read the documentation -- esp. about Bluemonday config.

[phtan]

@bep I agree that the documentation should be read.

But Blackfriday doesn't use Bluemonday, and as far as I can see, neither does Hugo. The documentation itself says:

Blackfriday itself does nothing to protect against malicious content. If you are dealing with user-supplied markdown, we recommend running blackfriday's output through HTML sanitizer such as Bluemonday.

https://gowalker.org/github.com/russross/blackfriday

Bluemonday just isn't included/imported into Hugo, so how do I even begin to configure it?

[dmitshur]

I'm wondering if there's a place where you list out the custom methods that you've written for the renderer. You know, to register the custom methods. Maybe that's what's missing

There is no such thing. You said it yourself that jww.ERROR.Printf gets printed if enabled, so the method does run.

But Blackfriday doesn't use Bluemonday, and as far as I can see, neither does Hugo.

Bluemonday was a guess. If it's not used by hugo, then it can't be the problem.

What does hugo use to sanitize output? Or does it not sanitize it at all, meaning <script> would be printed as <script> rather than getting filtered out (or printed as <script>)?

I suggest trying to add some print statements right after the markdown is rendered, before any further post-processing, and see if that works at preserving the task list HTML.

[bep]

Hugo doesn't do any sanitize other than what is done by Blackfriday and the Go template engine (and some URL magic). I assumed Blackfriday delegated to Bluemonday as it was mentioned here (which would not show up directly in Hugo imports).

But the code path here should be pretty easy to debug. If you cannot figure it out, I may lend you an eye if you can open a PR with your code (just mark the PR with In progress or something)

[phtan]

I don't know enough about the code-base to make the changes that are required for this issue. Let someone else take over the implementation of this issue.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.