How to apply a generic parser/renderer to all tokens?

[dwhieb]

I'm trying to implement markdown attributes (analogous to markdown-it-attrs).

I've already updated the tokens so that they have an additional attributes property. For example, this markdown:

This paragraph has **bold**{.bold} text.

...includes a token like this:

{
  attributes: {
    class: `bold`,
  },
  raw:  `**bold**`,
  text: `bold`,
  type: `bold`,
}

Now I'd like to postprocess the HTML from each token after its rendered, adding the attributes to the HTML. The token above should output:

<b class="bold">bold</b>

Because attributes can be added to any element, I'd like to use a generic parser/renderer that isn't restricted to a specific token type. Otherwise I'd have to overwrite all of the existing renderers.

Ideally I'd like a hook called postprocessAllTokens or something similar, but that functionality doesn't exist, and I can't find a workaround.

I've already looked at #3408, but that solution is also missing the final step I'm looking for, of turning the tokens into HTML.

[dwhieb]

@ivanjaros How did you wind up solving this?

[ivanjaros]

import parseAttrs from 'attributes-parser';

const attributes = {
  name: 'attributes',
  level: 'inline',
  start(src) { return src.match(/^\{.*\}/)?.index; },
  tokenizer(src, tokens) {
    const match = src.match(/^\{(.*)\}/);
    if (match && tokens.length > 0) {
      const last = tokens[tokens.length-1]
      if (['image'].includes(last.type)) {
        last.attributes = parseAttrs(match[1])
        return {
          type: 'attributes',
          raw: match[0],
        };
      }
    }
  }
};

This works with inline tokens(like image above), because it follows the markdown token and they are separate tokens, but it will NOT work with block tokens because their regex won't match the opening line. So you would have to write your own block tokens with attributes in mind, extract them and then call the original renderers.

edit: to alter the rendered html, you have to override the original renderer, or change the token to your own and essentially hijack it altogether or alter it to html token.

edit2: in the end i ended up only using lexer because i do my own rendering in Vue.

[UziTech]

Or if you want to do it the same way as markdown-it-attrs you will have to override the tokenizers for every token and add the attributes to the token.

[dwhieb]

I accomplished that using this extension:

import parseAttributes from 'attributes-parser'

const attrRegExp = /^\{(?<attributes>.+?)\}/v // Anchored to start of string

const tokenizeAttributes = {

  level: `inline`,

  name: `attributes`,

  renderer() {
    return ``
  },

  tokenizer(src, tokens) {

    const match = attrRegExp.exec(src)
    const token = tokens.at(-1)

    if (match && tokens.length) {

      token.attributes = parseAttributes(match.groups.attributes)

      return {
        raw:  match[0],
        type: `attributes`,
      }

    }

  },

}

But I'm still stuck on how to get the attributes into the actual HTML during the parsing/rendering step.

[UziTech]

The best way is probably to use the processAllTokens hook and set a property on each token if it has an attributes token after it. Then you will have to update each renderer to look for that property on the token and add the attributes to the output html.

[dwhieb]

The first part of your suggestion is actually already handled by the above code:

token.attributes = parseAttributes(match.groups.attributes)

I'm working on the second part and will report back on how it goes. Thanks for the guidance so far.

[dwhieb]

Here's my complete solution:

https://github.com/digitallinguistics/ling-md/tree/dd85058c7350b45339bd9d528eedc3e099075a27/extensions/attributes

• I have separate tokenizers for inline elements, block elements, and fenced code blocks.
• An attributes property is added to any token that has them.
• Each of the renderers are overwritten with a function that takes their output and uses the attributes property of the token to update the HTML.

Any feedback is most welcome. Once I've used this in production for a bit and ironed out any wrinkles that arise, I'll probably publish it as a Node package.

[calculuschild]

The Homebrewery has had an extension that basically does this for a few years now. One extension for block tokens and another for inline tokens. Supports appending HTML attributes, inline styles, or CSS classes.

Looks something like this:

![](pathToImage.png){width:100%,height:100%,customAttr='value',.customClass}

Probably needs a refactoring update now that Marked has released new hooks, but might be worth looking into how the extension is written there for inspiration.

Look for the "mustacheInjectInline" and "mustacheInjectBlock" extensions in this file: https://github.com/naturalcrit/homebrewery/blob/master/shared%2Fnaturalcrit%2Fmarkdown.js

[dwhieb]

Interesting, thank you. The inline solution is exactly the same approach I used at first, though eventually I decided against overriding the token's type property, and opted for a custom parser instead.

I hadn't thought about applying that some approach to block tokens though, that's interesting. Maybe I can simplify my own approach that way. Thanks!

[dwhieb]

Yup, worked like a charm, and simplified the codebase a bit too:

https://github.com/digitallinguistics/ling-md/tree/main/extensions/attributes