Passing information between transformers?

[emmerich]

I've got the following pipeline for parsing Markdown:

unified()
  .use(remarkParse)
  .use(remarkRehype, {
    allowDangerousHtml: true
  })
  .use(rehypeRaw)

Essentially passing through Rehype again to extract any embedded HTML in the Markdown. However, on larger Markdown documents, this is expensive and often unnecessary (a lot of Markdown does not have embedded HTML).

I want to modify remarkRehype so that, when it encounters an html node, it sets a "flag" on the processor which says "I've seen an HTML node, we need to run rehypeRaw". If the flag is not set, I skip rehypeRaw.

Using the current interface, we can imagine something like this:

unified()
  .use(remarkParse)
  .use(remarkRehype, {
    allowDangerousHtml: true,

    handlers: {
      html(h, node) {
        // set some sort of flag here
        return defaultHtmlHandler(h, node);
      },
    }
  })
  .use((opts) => {
        return function (tree) {
            if (flag) {
              return rehypeRaw(opts)(tree);
            }

            return tree;            
        };
    })

What's the best way of passing this flag between the processors?

Thanks!

[ChristianMurphy]

I want to modify remarkRehype so that, when it encounters an html node, it sets a "flag" on the processor which says "I've seen an HTML node, we need to run rehypeRaw". If the flag is not set, I skip rehypeRaw.

What are you trying to achieve here? This feels like an XY problem.
rehypeRaw scans for raw nodes, if it doesn't find any, then it doesn't do anything.
My guess is you are looking for a performance boost? If so, this likely wouldn't achieve your goal, skipping a scan here would have a negligible different on performance.

Passing information between transformers?

Unified uses the data attribute to pass around metadata.
This can be done at a file level https://github.com/vfile/vfile#vfiledata, using the second parameter of transformer https://github.com/unifiedjs/unified#function-transformernode-file-next

Or it can be done on individual AST nodes https://github.com/syntax-tree/unist#node

What's the best way of passing this flag between the processors?

I'm not sure what you mean by this. do you mean a transformer? or are there multiple instances of unified (a processor) on the page?

[emmerich]

I should clarify w.r.t. performance - negligible is a very subjective term.

rehype-raw (the highlighted anonymous function in the screenshot) is taking 2x longer than remark-rehype (toHast) to reparse the tree. In reality, we're talking about 100ms for a large Markdown document. But we're running our pipeline on 100,000s of documents and any small performance improvements we can find are invaluable.

This particular improvement has the benefit of being both impactful and very easy to implement/understand - we've already seen the tree during the remark-rehype phase and we know when we've marked a Node raw, so passing that information to the rehype-raw processor just makes a lot of sense.

Hope this helps to clear up our motivations

[wooorm]

Super cool to hear you’re using unified on such big datasets, Steven! Glad it works!

As you show with the 100ms, I believe hast-util-raw (edit: fixed typo) indeed passes everything through parse5, which will indeed be a useless and expensive operation if there’s no HTML.

For your Q on data: the .data function on processors (unified) is different than the .data on vfile, and also separate from the .data field on nodes!

[ChristianMurphy]

It's non-zero yes, and on a large enough scale it can matter.
But usually not the largest step, parsing typically is typically the most expensive part of a processor, and the new remark parser https://github.com/micromark/micromark likely will have a greater impact to improving times than transformer level updates.

I believe mdast-util-to-hast indeed passes everything through parse5, which will indeed be a useless and expensive operation if there’s no HTML.

Huh, I thought it just ran over the raw nodes, good to know.

[wooorm]

sorry, whoops, typo: mdast-util-to-hast is fast, hast-util-raw (in rehype-raw) does that*

[emmerich]

Yeah, I've been keeping an eye on micromark - looks good. From our side we need to improve our test coverage before we're ready to make the switch but it's definitely on our roadmap.

Parsing is another big fish in the performance breakdown, but given that micromark is on its way I've decided just to tackle some of the "low hanging fruit" that can easily improve things.