Richer rich-text format (Markdown?)

[andersk-auto]

Zephyr's current Scribe-derived formatting, apart from being a rather strange choice today, is somewhat limited in functionality. Some features that desktop or web clients almost certainly want to be able to render, and terminal clients may want to at least make an attempt at rendering, include

• URL links, either marking strings as URLs or marking non-URL strings as links to some other URL
• non-monospace fonts (see also Syntax for word-wrapping messages (format=flowed?) #117)
  • monospace fonts within non-monospace prose, for emphasis for command lines, etc.
  • monospace fonts within otherwise non-monospace text, for shell output, program code, etc.
• quoted replies (see also RFC 3676 as discussed in Syntax for word-wrapping messages (format=flowed?) #117, although that ticket isn't specifically about quoting replies)
• inline images (disabled by default, presumably, but useful at some sites)
• lists, bulleted and numbered
• superscript/subscript
• tagged references to usernames/people, so you have better options than substring-matching your username
• emoticons/emoji

Almost all of these are things that [http://daringfireball.net/projects/markdown/ Markdown] supports. The official spec is unmaintained and has some widely-acknowledged deficiencies, and is generally forked one way or another by implementors, which we'll likely have to do. For instance, there's [https://help.github.com/articles/github-flavored-markdown GitHub-Flavored Markdown], which notably includes @-mentions, as well as better handling for code (which seems useful, given Zephyr's current user base).

I understand that Zulip has some formatting syntax, based on Markdown, that's worth considering here. I'm not sure if there's anything Zulip supports that I didn't cover above, but it'd probably be nice to make sure that, preferably, we settle on a syntax Zulip is happy with, or failing that, we find something that can losslessly round-trip.

It might also be worth looking at other IM or group chat protocols (AIM, Jabber, gchat, !HipChat, etc.) to see if there's anything useful they support that's not on the list above.

I don't really think it makes sense to retrofit this into existing Scribe-based syntax (I certainly would much prefer to use something based on a current language, and a combination of Scribe and a more recent wikitext just seems like a poor plan), so I think we should find some syntax for signaling that an entire zephyrgram is in Markdown. One option that occurs to me is to use the default format for this, and have it include the magic string @mdwn($2) (which is at least vaguely useful to indicate to old clients what's going on, even though it's just treated as magic by new clients, and @mdwn isn't actually defined). Markdown as sent on the wire should be zephyr-escaped (i.e., double at-signs), so that Markdown-unaware clients don't see rendering issues and see raw Markdown -- which is fine, and part of Markdown's intended purpose. It doesn't seem worthwhile to try to generate a Scribe-style rendering of the Markdown.

We should then add something to clients to enable composing in Markdown by default, with some option for reverting to Scribe-style markup. Clients should be capable of rendering both. (An overly-clever way to make the latter easy would be to add a Zephyr library function to parse Scribe-style markdown and emit Markdown.)

It's worth taking into account what syntax is likely to get triggered by accident. I think Markdown has relatively little of this if you take out intra-word underscores (as GFM does).

Replace "Markdown" everywhere with whatever wikitext we end up using, if it isn't Markdown.

[andersk-auto]

Imported from trac issue 120. Created by geofft@1TS.ORG on 2013-08-10T23:41:13, last modified: 2013-08-30T06:39:33

[andersk-auto]

Trac comment by geofft@1TS.ORG on 2013-08-11 00:30:59:

It's also worth noting that there should be a client mode in zwrite(1) to send easily plain text, either wrapped monospace plain text or preformatted text, and a recommendation that other clients do the same. We already have problems with at-signs (I think this is barnowl:ticket:122) and Markdown risks making this somewhat worse.

[andersk-auto]

Trac comment by andersk@ATHENA.MIT.EDU on 2013-08-30 06:39:33:

Markdown or anything like it is completely worthless as an interchange format for interoperability. The entire idea of trying to shoehorn a rough collection of conflicting human input conventions into a rigid computer-readable (to say nothing of computer-writable) interchange format is just completely insane. In case you don’t believe that already, I present Exhibit A.

(Keep in mind that Markdown’s problems don’t preclude its use as an ''input'' format. There is no reason the input format can’t be totally decoupled from the interchange format, which makes the input format a client-side user interface problem. The sender will notice if their message is misrendered, as long all recipients see it misrendered in the same way.)

I see two better options:

• Use HTML5 or XHTML5 as the interchange format. Both have well-specified parsing and error-handling behavior and good library support across many languages. I’d recommend stipulating white-space: pre-wrap globally so we don’t have to deal with whitespace collapsing annoyances. We’d have to be a little careful to make sure cross-realm scripting attacks don’t become a thing.
• Keep the existing scribe format, extending it as necessary. Existing clients display unknown markup literally, so we get graceful backwards compatibility for free. As far as word-wrapping is concerned, geofft and I independently came up with the idea of using empty groups @() to mark soft line breaks, so maybe that’s not entirely crazy.

I think I have a slight preference for the latter. It sure sounds like a lot less work.