HTML translation with Argos Translate

[PJ-Finlay]

https://github.com/argosopentech/translate-html

[pierotofy]

Any particular area that should be tested (or particular inputs?). Or just general feedback?

[pierotofy]

Some general feedback based on test inputs for the HTML translate part:

1. First, I feel like this is performing quite well! (minus some problems, more below)

2. Spaces seem to get obliterated when parsing HTML paragraphs:

test.zip

Input:
Output:

3. Parsing more complex HTML pages invalidates parts that should not be translated (this is expected at this point, but I guess thoughts for smarter training). In particular anything in between <script></script> gets translated, which leads to a bit of a mess. I'm not even sure this is a real issue, as a smarter HTML parsing could avoid those parts already. There's a lot of HTML tags that could be part of a "blacklist" of "do-not-translate" (stylesheets, canvas elements, pre/code sections?, etc.).

4. As an extention of point 3, it certainly seems the case that HTML-specific semantics might be necessary to increase translation quality. For example, anchor (<a>) tags have certain attributes (href, style, class, etc.) that should never be translated (which don't seem to be), but others do (title).

<a class="mw-redirect" href="/wiki/Razze" title="Lista di cento specie invasive molto dannose">breeds</a>different recognized with certifications. Essentially territorial and dusk, the cat is a<a class="mw-redirect" href="/wiki/Predatore" title="Predatore">predator</a>

(Translated from Italian to English, note how title remain in Italian, but a better parser would translate the title since it's meant for end-users to read, if one hovers over a link.

[PJ-Finlay]

The current implementation should only translate content in tags but not tag metadata. Differentiating between tag types probably makes sense though. Thoughts on a blacklist vs a white list?

[PJ-Finlay]

Losing spaces is a know problem, I thought I fixed it but I'll take another look.

[PJ-Finlay]

For the core library including more html semantics like link titles in the metadata probably doesn't make sense but there's lots of room for application specific parsing built on top.

[PJ-Finlay]

Looking at a list of tags I think a blacklist of script and a few others will make sense.

[pierotofy]

👍 on the blacklist.

Agree that HTML semantics are probably something to build on top.

[PJ-Finlay]

I fixed the whitespace formatting issue and added a blacklist. Planning to release this weekend.

Translations are looking good!

[PJ-Finlay]

It appears tag injection is rarely used but just translating tags individually seems to do a reasonable job in a lot of cases.

[PalmerAL]

Hey, I just saw this discussion. I just released something similar for my project, which I think you saw. I'm wondering how this works based on your last comment - is it doing tag injection, or is it translating tags individually?

I took the approach of translating individual tags, and it does generally work pretty well. However, it tends to fail when there's a tag near the end of a sentence, and you get a one or two-word chunk of text that's in its own node. When that happens, the translation engine sometimes produces totally nonsensical results. For example, on this page: https://www.grochtdreis.de/

dass auch Aspekte der <strong>Barrierefreiheit</strong>, <strong>Usability</strong>, <strong>Performance</strong> und <strong>Suchmaschinenoptimierung</strong> berücksichtigt werden.</p>

"berücksichtigt werden" on its own is translated into "The Commission will be able to take account of the situation in the Member States".

There were two heuristics I implemented that helped avoid some issues:

1. If the node contains only numbers, skip translating it, because the model doesn't handle non-word inputs well.
2. If the length of the translated text is >20x the original length, skip the translation and keep the original.

I'm not sure if any of that's helpful, or if you've already done that, but I figured I'd mention it in case.

[PJ-Finlay]

Oh cool, I saw you had integrated LibreTranslate but didn't realize you had HTML translation. Those are pretty reasonable heuristics.

I implemented tag injection but it can rarely inject successfully so it's essentially translating individual tags. All I did was blacklist some tags like script and code that shouldn't be translated.

[pujianto]

Hi,
Happy holiday & thank you for the useful software.
For speedy html parsing, Is this possible to use lxml parser instead of html.parser?

Thanks

reference: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser

[PJ-Finlay]

There's no interface (but could add one if needed), here's the line that sets the parser.

[ghost]

Thank you very much for this tool. It came in extremely useful, and the resulting translation was very impressive. I exported HTML from an EPUB book using Sigil, and translated the whole book from French to English and it came out very readable.

I could see myself using this a lot. The only issue I had is that it was rather slow. The whole book took about 45 minutes to translate. Would multi threading help? I didn't see a big spike in CPU, it just ran slow for some reason. Not sure what the bottleneck is

EPUB translators are very hard to find, so this was super handy.

[PJ-Finlay]

Glad it's working for you!

CTranslate2 uses multiple threads to do translations but by default Argos Translate will translate the document sequentially. If you want translation to go faster you could try using more powerful hardware or splitting the document into sections and translating in parallel.

[dingedi]

Added support for .epub
https://github.com/dingedi/argos-translate-files/releases/tag/v1.1.0

[VVVV-2]

When translating single words within a tag (h, strong, i, etc..) that could be considered both Swedish and English words ('Start', 'test') I get this word returned multiple times. <h1>Test</h1>
becomes
<h1>Test test test test test test test test test test test test test test test test test test test test</h1>
when going "sv" -> "en".

Can anyone point me in the right direction of what to look for?