Extract text more carefully in `mdbook-xgettext`

[mgeisler]

Right now, we simply split the text on \n\n+, but this leads to a number of problems:

• We split code blocks into different messages when there are one or more blank lines in the middle of the block.
• We extract bullet point lists as a single message.

In general, it would be awesome if we could

• Make the extracted messages independent of the precise formatting of the Markdown text. In particular, a hard-wrapped paragraph should be extracted without the paragraph breaks.
• Remove formatting such as # from headers and * from bullet points.
• Extract code blocks as a single message.

So Markdown like

# This is a heading

A _little_
paragraph.

```rust,editable
fn main() {
    println!("Hello world!");
}
```

* First
* Second

should result in these messages

• This is a heading (heading type is stripped)
• A _little_ paragraph. (softwrapped lines are unfolded)
• fn main() {\n println!("Hello world!");\n} (info string is stripped)
• First (bullet point extracted individually)
• Second

You could imagine done something nice with links too: foo [bar](https://example.net) baz could be stored as foo [bar] baz. This might be a poor idea, though: it means that the translator cannot change the destination URL.

[mgeisler]

When doing this, it's critically important that we run the same transformations on the existing po/*.po files. That way we can keep the work of the translators intact.

[moutikabdessabour]

why not use the markdown parser used by mdbook pulldown_cmark to extract paragraph and then reconstruct it?

[mgeisler]

@moutikabdessabour we should definitely use pulldown_cmark for this!

[djmitche]

I'd like to work on this, if you don't mind assigning it to me. I can see how to replace the existing extract_paragraphs with a more sophisticated thing that emits a bunch of textual chunks.

run the same transformations on the existing po/*.po files

Did you have something "easy" in mind for this? My thinking was that this would be a kind of half-automated process, where with some iteration I could find a one-off way to translate all of the old msgid's to new msgid's, and then apply those to the po/*.po files, keeping the existing translation.

[jooyunghan]

While working on the Korean translation i found that keeping MD stuff(ie bullets) was helpful because it gave me freedom to do whatever fits better in the target language like splitting a single bullet into two when necessary. It's probably the same reason why mgeisler@ thought trimming links would be a poor idea.

[djmitche]

That makes a lot of sense. I think we could adjust the chunk-extraction to collapse adjacent list-item chunks into a single chunk.