Refactor HTML Parser

.spell-dict

@@ -131,6 +131,7 @@ Treeprocessor
 Treeprocessors
 tuple
 tuples
+unclosed
 unescape
 unescaping
 unittest

docs/change_log/release-3.3.md

@@ -66,6 +66,21 @@ The following new features have been included in the 3.3 release:
   Any random HTML attribute can be defined and set on the `<code>` tag of fenced code
   blocks when the `attr_list` extension is enabled (#816).
 
+* The HTML parser has been completely replaced. The new HTML parser is built on Python's
+  [html.parser.HTMLParser](https://docs.python.org/3/library/html.parser.html), which
+  alleviates various bugs and simplify maintenance of the code (#803, #830).
+
+* The [Markdown in HTML](../extensions/md_in_html.md) extension has been rebuilt on the
+  new HTML Parser, which drastically simplifies it. Note that raw HTML elements with a
+  `markdown` attribute defined are now converted to ElementTree Elements and are rendered
+  by the serializer. Various bugs have been fixed (#803, #595, #780, and #1012).
+
+* Link reference parsing, abbreviation reference parsing and footnote reference parsing
+  has all been moved from `preprocessors` to `blockprocessors`, which allows them to be
+  nested within other block level elements. Specifically, this change was necessary to
+  maintain the current behavior in the rebuilt Markdown in HTML extension. A few random
+  edge-case bugs (see the included tests) were resolved in the process (#803).
+
 ## Bug fixes
 
 The following bug fixes are included in the 3.3 release:

docs/extensions/md_in_html.md

@@ -4,122 +4,234 @@ title: Markdown in HTML Extension
 
 ## Summary
 
-An extensions that parses Markdown inside of HTML tags.
+An extension that parses Markdown inside of HTML tags.
 
-## Usage
+## Syntax
 
-From the Python interpreter:
+By default, Markdown ignores any content within a raw HTML block-level element. With the `md-in-html` extension
+enabled, the content of a raw HTML block-level element can be parsed as Markdown by including  a `markdown` attribute
+on the opening tag. The `markdown` attribute will be stripped from the output, while all other attributes will be
+preserved.
 
-```pycon
->>> import markdown
->>> html = markdown.markdown(text, extensions=['md_in_html'])
-```
+The `markdown` attribute can be assigned one of three values: [`"1"`](#1), [`"block"`](#block), or [`"span"`](#span).
 
-Unlike the other Extra features, this feature is built into the markdown core and
-is turned on when `markdown.extensions.extra` or `markdown.extensions.md_in_html`
-is enabled.
+!!! note
 
-The content of any raw HTML block element can be Markdown-formatted simply by
-adding a `markdown` attribute to the opening tag. The markdown attribute will be
-stripped from the output, but all other attributes will be preserved.
+    The expressions "block-level" and "span-level" as used in this document refer to an element's designation
+    according to the HTML specification. Whereas the `"span"` and `"block"` values assigned to the `markdown`
+    attribute refer to the Markdown parser's behavior.
 
-If the markdown value is set to `1` (recommended) or any value other than `span`
-or `block`, the default behavior will be executed: `p`,`h[1-6]`,`li`,`dd`,`dt`,
-`td`,`th`,`legend`, and `address` elements skip block parsing while others do not.
-If the default is overridden by a value of `span`, *block parsing will be skipped*
-regardless of tag. If the default is overridden by a value of `block`,
-*block parsing will occur* regardless of tag.
+### `markdown="1"` { #1 }
 
-#### Simple Example:
+When the `markdown` attribute is set to `"1"`, then the parser will use the default behavior for that specific tag.
 
-```md
-This is *true* markdown text.
+The following tags have the `block` behavior by default: `address`, `article`, `aside`, `blockquote`, `body`,
+`colgroup`, `details`, `div`, `dl`, `fieldset`, `figcaption`, `figure`, `footer`, `form`, `iframe`, `header`, `hr`,
+`main`, `menu`, `nav`,  `map`, `noscript`, `object`, `ol`, `section`, `table`, `tbody`, `thead`, `tfoot`, `tr`, and
+`ul`.
 
+For example, the following:
+
+```
 <div markdown="1">
-This is *true* markdown text.
+This is a *Markdown* Paragraph.
 </div>
 ```
 
-#### Result:
+... is rendered as:
 
-```html
-<p>This is <em>true</em> markdown text.</p>
+``` html
 <div>
-<p>This is <em>true</em> markdown text.</p>
+<p>This is a <em>Markdown</em> Paragraph.</p>
 </div>
 ```
 
-### Nested Markdown Inside HTML Blocks
+The following tags have the `span` behavior by default: `address`, `dd`, `dt`, `h[1-6]`, `legend`, `li`, `p`, `td`,
+and `th`.
 
-Nested elements are more sensitive and must be used cautiously. To avoid
-unexpected results:
+For example, the following:
 
-* Only nest elements within block mode elements.
-* Follow the closing tag of inner elements with a blank line.
-* Only have one level of nesting.
+```
+<p markdown="1">
+This is not a *Markdown* Paragraph.
+</p>
+```
 
-#### Complex Example:
+... is rendered as:
 
-```md
-<div markdown="1" name="Example">
+``` html
+<p>
+This is not a <em>Markdown</em> Paragraph.
+</p>
+```
 
-The text of the `Example` element.
+### `markdown="block"` { #block }
 
-<div markdown="1" name="DefaultBlockMode">
-This text gets wrapped in `p` tags.
-</div>
+When the `markdown` attribute is set to `"block"`, then the parser will force the `block` behavior on the contents of
+the element so long as it is one of the `block` or `span` tags.
 
-The tail of the `DefaultBlockMode` subelement.
+The content of a `block` element is parsed into block-level content. In other words, the text is rendered as
+paragraphs, headers, lists, blockquotes, etc. Any inline syntax within those elements is processed as well.
 
-<p markdown="1" name="DefaultSpanMode">
-This text *is not* wrapped in additional `p` tags.
-</p>
+For example, the following:
 
-The tail of the `DefaultSpanMode` subelement.
+```
+<section markdown="block">
+# A header.
 
-<div markdown="span" name="SpanModeOverride">
-This `div` block is not wrapped in paragraph tags.
-Note: Subelements are not required to have tail text.
-</div>
+A *Markdown* paragraph.
 
-<p markdown="block" name="BlockModeOverride">
-This `p` block *is* foolishly wrapped in further paragraph tags.
-</p>
+* A list item.
+* A second list item.
 
-The tail of the `BlockModeOverride` subelement.
+</section>
+```
+
+... is rendered as:
+
+``` html
+<section>
+<h1>A header.</h1>
+<p>A <em>Markdown</em> paragraph.</p>
+<ul>
+<li>A list item.</li>
+<li>A second list item.</li>
+</ul>
+</section>
+```
+
+!!! warning
+
+    Forcing elements to be parsed as `block` elements when they are not by default could result in invalid HTML.
+    For example, one could force a `<p>` element to be nested within another `<p>` element. In most cases it is
+    recommended to use the default behavior of `markdown="1"`. Explicitly setting `markdown="block"` should be
+    reserved for advanced users who understand the HTML specification and how browsers parse and render HTML.
+
+### `markdown="span"` { #span }
+
+When the `markdown` attribute is set to `"span"`, then the parser will force the `span` behavior on the contents
+of the element so long as it is one of the `block` or `span` tags.
+
+The content of a `span` element is not parsed into block-level content. In other words, the content will not be
+rendered as paragraphs, headers, etc. Only inline syntax will be rendered, such as links, strong, emphasis, etc.
+
+For example, the following:
 
-<div name="RawHtml">
-Raw HTML blocks may also be nested.
+```
+<div markdown="span">
+# *Not* a header
 </div>
+```
 
+... is rendered as:
+
+``` html
+<div>
+# <em>Not</em> a header
 </div>
+```
+
+### Ignored Elements
+
+The following tags are always ignored, regardless of any `markdown` attribute: `canvas`, `math`, `option`, `pre`,
+`script`, `style`, and `textarea`. All other raw HTML tags are treated as span-level tags and are not affected by this
+extension.
+
+### Nesting
 
-This text is after the markdown in HTML.
+When nesting multiple levels of raw HTML elements, a `markdown` attribute must be defined for each block-level
+element. For any block-level element which does not have a `markdown` attribute, everything inside that element is
+ignored, including child elements with `markdown` attributes.
+
+For example, the following:
+
+```
+<article id="my-article" markdown="1">
+# Article Title
+
+A Markdown paragraph.
+
+<section id="section-1" markdown="1">
+## Section 1 Title
+
+<p>Custom raw **HTML** which gets ignored.</p>
+
+</section>
+
+<section id="section-2" markdown="1">
+## Section 2 Title
+
+<p markdown="1">**Markdown** content.</p>
+
+</section>
+
+</article>
 ```
 
-#### Complex Result:
+... is rendered as:
 
 ```html
-<div name="Example">
-<p>The text of the <code>Example</code> element.</p>
-<div name="DefaultBlockMode">
-<p>This text gets wrapped in <code>p</code> tags.</p>
+<article id="my-article">
+<h1>Article Title</h1>
+<p>A Markdown paragraph.</p>
+<section id="section-1">
+<h2>Section 1 Title</h2>
+<p>Custom raw **HTML** which gets ignored.</p>
+</section>
+<section id="section-2">
+<h2>Section 2 Title</h2>
+<p><strong>Markdown</strong> content.</p>
+</section>
+</article>
+```
+
+When the value of an element's `markdown` attribute is more permissive that its parent, then the parent's stricter
+behavior is enforced. For example, a `block` element nested within a `span` element will be parsed using the `span`
+behavior. However, if the value of an element's `markdown` attribute is the same as, or more restrictive than, its
+parent, the the child element's behavior is observed. For example, a `block` element may contain either `block`
+elements or `span` elements as children and each element will be parsed using the specified behavior.
+
+### Tag Normalization
+
+While the default behavior is for Markdown to not alter raw HTML, as this extension is parsing the content of raw HTML elements, it will do some normalization of the tags of block-level elements. For example, the following raw HTML:
+
+```
+<div markdown="1">
+<p markdown="1">A Markdown paragraph with *no* closing tag.
+<p>A raw paragraph with *no* closing tag.
 </div>
-<p>The tail of the <code>DefaultBlockMode</code> subelement.</p>
-<p name="DefaultSpanMode">
-This text <em>is not</em> wrapped in additional <code>p</code> tags.</p>
-<p>The tail of the <code>DefaultSpanMode</code> subelement.</p>
-<div name="SpanModeOverride">
-This <code>div</code> block is not wrapped in paragraph tags.
-Note: Subelements are not required to have tail text.</div>
-<p name="BlockModeOverride">
-<p>This <code>p</code> block <em>is</em> foolishly wrapped in further paragraph tags.</p>
+```
+
+... is rendered as:
+
+``` html
+<div>
+<p>A Markdown paragraph with <em>no</em> closing tag.
+</p>
+<p>A raw paragraph with *no* closing tag.
 </p>
-<p>The tail of the <code>BlockModeOverride</code> subelement.</p>
-<div name="RawHtml">
-Raw HTML blocks may also be nested.
 </div>
+```
 
-</div>
-<p>This text is after the markdown in HTML.</p>
+Notice that the parser properly recognizes that an unclosed  `<p>` tag ends when another `<p>` tag begins or when the
+parent element ends. In both cases, a closing `</p>` was added to the end of the element, regardless of whether a
+`markdown` attribute was assigned to the element.
+
+To avoid any normalization, an element must not be a descendant of any block-level element which has a `markdown`
+attribute defined.
+
+!!! warning
+
+    The normalization behavior is only documented here so that document authors are not surprised when their carefully
+    crafted raw HTML is altered by Markdown. This extension should not be relied on to normalize and generate valid
+    HTML. For the best results, always include valid raw HTML (with both opening and closing tags) in your Markdown
+    documents.
+
+## Usage
+
+From the Python interpreter:
+
+``` pycon
+>>> import markdown
+>>> html = markdown.markdown(text, extensions=['md_in_html'])
 ```

markdown/blockprocessors.py

@@ -51,6 +51,7 @@ def build_block_parser(md, **kwargs):
     parser.blockprocessors.register(OListProcessor(parser), 'olist', 40)
     parser.blockprocessors.register(UListProcessor(parser), 'ulist', 30)
     parser.blockprocessors.register(BlockQuoteProcessor(parser), 'quote', 20)
+    parser.blockprocessors.register(ReferenceProcessor(parser), 'reference', 15)
     parser.blockprocessors.register(ParagraphProcessor(parser), 'paragraph', 10)
     return parser
 
@@ -554,6 +555,35 @@ def run(self, parent, blocks):
             )
 
 
+class ReferenceProcessor(BlockProcessor):
+    """ Process link references. """
+    RE = re.compile(
+        r'^[ ]{0,3}\[([^\]]*)\]:[ ]*\n?[ ]*([^\s]+)[ ]*\n?[ ]*((["\'])(.*)\4|\((.*)\))?[ ]*$', re.MULTILINE
+    )
+
+    def test(self, parent, block):
+        return True
+
+    def run(self, parent, blocks):
+        block = blocks.pop(0)
+        m = self.RE.search(block)
+        if m:
+            id = m.group(1).strip().lower()
+            link = m.group(2).lstrip('<').rstrip('>')
+            title = m.group(5) or m.group(6)
+            self.parser.md.references[id] = (link, title)
+            if block[m.end():].strip():
+                # Add any content after match back to blocks as separate block
+                blocks.insert(0, block[m.end():].lstrip('\n'))
+            if block[:m.start()].strip():
+                # Add any content before match back to blocks as separate block
+                blocks.insert(0, block[:m.start()].rstrip('\n'))
+            return True
+        # No match. Restore block.
+        blocks.insert(0, block)
+        return False
+
+
 class ParagraphProcessor(BlockProcessor):
     """ Process Paragraph blocks. """