Creole reader

[swilde]

I wrote a reader for Creole 1.0 http://www.wikicreole.org/wiki/Creole1.0 including a bunch of tests.
It is feature complete, not very thoroughly tested yet, though.

[jgm]

Fantastic!

Have you tested this much on real-world creole? I tried it out on the source of http://www.wikicreole.org/wiki/Creole1.0 (which I'm assuming is creole), and I noticed some
differences between the HTML pandoc generated and the page that displays on the wiki.

For example:

• some of the code samples are showing up as inline code, where they have code blocks on the wiki
• Sample output: "italics" (instead of actually being in italics)
• Missing space: "ableto cross paragraphs"
• Missing emphasis in "__blog-style vs. wiki-style__", "__must__"
• [long discussion|Talk.Change Linebreak Markup Proposal] not linkified
• Sample output for table doesn't show up as a table (maybe because there's not a blank line before it?)

I'd suggest running this against a number of real-world creole texts and comparing your parser's output to the output of a standard creole parser. (I note that the creole wiki's sandbox says it's not a reference implementation of creole -- so maybe the above are bugs in it rather than in your reader -- but presumably there is a reference implementation somewhere!)

What you have looks close to being ready to merge, and if some of the kinks can be ironed out soon enough, we might even be able to include this in the upcoming 2.0 release.

[jgm]

PS. In addition to the highly focused tests you have, you might add some tests with multiple blocks, since it's important to test the transitions between blocks. Or you could add an "old" style reader (golden) test, like mediawiki-reader.wiki/mediawiki-reader.native (put files in test/ and add to Test.Pandoc.Old), to supplement the tests you have.

[swilde]

Thanks for your reply.

Have you tested this much on real-world creole?

Yes. I tried it with a few creole documents from the company I'm
working for.

I tried it out on the source of
http://www.wikicreole.org/wiki/Creole1.0 (which I'm assuming is
creole), and I noticed some differences between the HTML pandoc
generated and the page that displays on the wiki.

I'll check them out and make fixes where I think they are appropriate.
(Might take some time, this is a spare time project of mine).

I'd suggest running this against a number of real-world creole texts
and comparing your parser's output to the output of a standard
creole parser. (I note that the creole wiki's sandbox says it's not
a reference implementation of creole -- so maybe the above are bugs
in it rather than in your reader -- but presumably there is a
reference implementation somewhere!)

There is no reference implementation I'm aware of, and to my best
knowledge there is no official reference implementation. Quite
contrary: the implementation of the official wiki is known to be not
compliant to Creole 1.0.

So the only official reference for Creole 1.0 is the textual
description on http://www.wikicreole.org/wiki/Creole1.0
(which is is kind of "wobbily" as all informal specifications by
example tend to be).

As a reference test case I used creole1.0test.txt from
http://www.wikicreole.org/wiki/Creole1.0TestCases as pointed out on
the site the text file is preferred as to avoid onfusion caused by
bugs in this wiki engine.

What you have looks close to being ready to merge, and if some of
the kinks can be ironed out soon enough, we might even be able to
include this in the upcoming 2.0 release.

Thant really sounds great. Thanks!

[swilde]

Hi,

I looked into the problems you reported:

• some of the code samples are showing up as inline code, where they
  have code blocks on the wiki

This is the case e.g. with the description of bold and italic. But
I'd argue that my rendering is correct.

For example, the wiki source reads

Creole:
{{{ **bold** }}}

Recommended XHTML:
{{{ <strong>bold</strong> }}}

The standard says on NoWiki (reformatted, verbatim text):

This works inline or as a block. [...] As a block, the three curly
braces should be on one line by itself to open and another line of
three curly braces should be on a line by itself to close, without
leading spaces.

So {{{ **bold** }}} is inline nowiki, as a block it would have to
be:

{{{
**bold**
}}}

• Sample output: "italics" (instead of actually being in italics)

It's ''italics'' to be precise, and that's simply what is in the wiki
source. This is no valid Creole markup, it is the Wiki Syntax used by
JSPWiki, which is the base technology used on the Wiki.

Same is true for __bold__.

• Missing space: "ableto cross paragraphs"

Right. I fixed this.

• Missing emphasis in "__blog-style vs. wiki-style__", "__must__"

Once again, this is not Creole but JSPWiki markup.

• [long discussion|Talk.Change Linebreak Markup Proposal] not linkified

Same here: Creole link would be [[foo bar|Link Name here]] (double brackets).

• Sample output for table doesn't show up as a table (maybe because
  there's not a blank line before it?)

There are two issues:

1. Once again it's no valid Creole, table headers are marked with |=
   not || in Creole.

2. You are right, my parser required a blank line to separate tables
   from preceding paragraphs. I changed that now.

Regarding the additional Tests. I'll look into that the next days...

What else would be required for the reader to get merged?

cheers
sascha

[jgm]

I think it's already good to merge. More tests would be good, but we can merge those in later.

[jgm]

Thank you!