Various Extension System Problems

[mitsuhiko]

I just looked at how the grammar is extended and in it's current form it's not really usable. There are a few problems with the way the inline lexer is supposed to modified alone:

• right now the example modifies default_rules which is a class attribute and as such shared across all other lexers as well.
• because the text rule is greedy it needs to be modified whenever another rule is inserted but it is impossible for a plugin to do so because it cannot now which sequences indicate the start of a rule. The current example only works by chance.

I think the current design does not really suit itself to extension well but I'm not sure how to fix it without breaking what's already there.

[lepture]

It was first ported from marked.js. This is why it follows the design of marked. I've noticed it is hard to be extended #62

[mitsuhiko]

Yeah. It's quite disappointing :( If you want some more concrete advise on how to do it I can probably outline some general suggestions.

[lepture]

@mitsuhiko That would be helpful. I am not good at writing parsers.

[mitsuhiko]

Goddammit :) Do you have some text messenger? Easier than issue trackers for this sort of stuff.

[lepture]

@mitsuhiko How about https://telegram.me/lepture

[mitsuhiko]

I investigated the re module a bit for that purpose and discovered how the undocumented re.Scanner is implemented. A modified version of it is perfect for mistune: https://gist.github.com/mitsuhiko/153c29b778068d3798fc

[nitely]

@lepture @mitsuhiko Any progress on this?

I've been toying around with @mitsuhiko's scanner and got a basic parser working (no extensions yet). The only issue so far is backreferences seem to not work, so it's probably not possible to avoid breaking changes in mistune.

[lepture]

@nitely not yet.

[nitely]

Hey @lepture ,

I'm don't know how advanced this re-implementation is, but I wanted to share what I have so far[0], since I'm not planning to ever releasing it as a library. It might serve you as inspiration.

Every element is implemented as an extension, so it should be highly extensible. The text is parsed, then an AST is generated, then the AST is rendered as HTML. A different render can be easily implemented as well.

I've written it a while ago, but IIRC, it's able to correctly parse daring fireball examples and docs. Benchmarks shows it's as fast as mistune.

[0] https://gist.github.com/nitely/9fe34f64cd300aaa92af731f5634ce8b

[lepture]

@nitely Thanks for sharing. Scanner is different in Python 2 and Python 3, that's why I'm hesitated to implement it in scanner. Currently my focus is on other projects, so that this will be hold on for a while.

[nitely]

Thanks for sharing.

NP 😺

Scanner is different in Python 2 and Python 3, that's why I'm hesitated to implement it in scanner.

mmh it is. But still works in py2 and py3 (the code I shared is for py3). I made a PR for fixing the original @mitsuhiko code mitsuhiko/python-regex-scanner#3 .

Currently my focus is on other projects, so that this will be hold on for a while.

Yeah, no problem. mistune is actually good enough for me at its current state.

[lepture]

v2 is ready now.