Improve the API script to handle | characters

[arnaucasau]

Summary

This PR changes how we handle pipes inside math expressions when we convert sphinx HTML into markdown. Some math expressions use pipes to define quantum states using the Dirac notation, and we need to escape those characters to avoid breaking the page when the pipe characters are used inside a markdown table.

Details

One solution could only handle the | characters used inside a table, but given that the math expressions could be used in nested tags (e.g <td> <p> <span class="math"> SOME_EXPRESSION </span></p></td>), and would need to make the script more complex without ensuring we fix all the cases where we could make the page to fail to render, I decided to handle that character differently in all math expressions.

The PR replaces the | character with \vert which will represent the same character. \vert needs extra space at the end to handle cases where the pipe was next to a non-numerical character. In those cases, we should avoid converting |x to \vertx given that the latter is not a valid command (\vert x is the correct conversion).

We also need to take into account that, in some cases, we still need to use the | characters because when escaped (\|), it represents a double pipe (||), which could be used in different mathematical expressions like the length of a vector.

This is the regex used, which only matches pipe characters not preceded by a backslash:

/(?<!\\)\|/gm

A new test was added to verify different cases where we can find a pipe character. The tests checks | characters outside math expressions, in math expressions inside a table, and in math expressions outside the table. We can also find a case where we intentionally want to use \| to create a double pipe. In the following screenshot, we can see the rendered result of the test.

Closes #488

[frankharkins]

Nice work! I've re-generated the docs with the new script and it all LGTM 🚀

[Eric-Arellano]

Unfortunately, this resulted in some pages crashing, per #673 (comment). This is because we broke column alignment settings for matrices, like this from PauliTable.md:

\begin{split}\left(\begin{array}{ccc|ccc}

If you use \vert for {ccc|ccc}, you get:

ParseError: KaTeX parse error: Unknown column alignment: \vert at position 37: …egin{array}{ccc\̲v̲e̲r̲t̲ ̲ccc}
    x_{0,0…

Maybe we were over-zealous with always replacing | with \vert. What about we fix #488 solely be fixing the problematic line? It's the only instance of this problem we've found! We actually could now even fix this by changing the HTML directly to use \vert. That works because these are historical API docs so we can modify the input HTML thanks to #737. The APIs were removed from modern Qiskit, so no worries about fixing qiskit/qiskit.