Multi-lingual Language/Swapping

[PhilienTaylor]

Hi there,

As I write and contemplate tools that can help democracy, I was introduced to Sideband through the Reticulum chatroom.

Would it be possible to begin multi-language support by offloading language, labels, and other words into a file for easy cloning and region-zoning?

I wonder if a Creole, Hebrew, Arabic, Chinese, Spanish, and other languages on and on, might help the world. 🙇‍♀️ Thank you.

[markqvist]

It would be really great to have this, but it's not something I personally have the resources to do currently. If the community wants to take up this task, it would be great!

I don't think it's going to be a small effort though, since there's a lot of strings and labels scattered around, but if anyone has the courage, I would be grateful!

Another approach entirely, would be to create a new mobile-centric LXMF messenger, focused on the basics first, with a fast UI (using native Android UI elements, for example), that has multi-language support from the beginning.

Many of the problems with internationalization of Sideband stems from the fact that the Kivy/KivyMD frameworks used for rendering the UI are not very good at rendering international text, currently.

[PhilienTaylor]

Thank you, Mark. A question I have is that I saw a multilingual library being loaded; are you familiar with it enough to talk about how it might provide the language string formatting, global label spaces, and other spaces? As a note, if you know enough on how to begin the process, you might provide translators enough space to introduce Reticulum and Sideband to other international developers. Thank you for all your work! :) I'll send my prayers out to the community! 🙇‍♀️

…

On Wed, Mar 20, 2024, 6:23 AM markqvist ***@***.***> wrote: It would be really great to have this, but it's not something I personally have the resources to do currently. If the community wants to take up this task, it would be great! I don't think it's going to be a small effort though, since there's *a lot* of strings and labels scattered around, but if anyone has the courage, I would be grateful! — Reply to this email directly, view it on GitHub <#468 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5EA5M3XUBMDZWQTE2PFEG3YZFPS7AVCNFSM6AAAAABE6QSK5WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQNJQHE4TK> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[markqvist]

Unfortunately, the only thing I have sufficient experience with so far, in that regards, is the multilingual text input which was implemented a little while back. Messages can now be written and displayed in many different scripts, so that is at least something.

But the actual UI elements, labels etc., is not something I am familiar enough with to provide any concrete pointers. The general idea is to simply provide a references to a multilingual string instance everywhere that multilingual text needs to be rendered, and let that class return the right string for the currently selected language. I am sure there is already good implementations available for this in Python, but I don't have enough experience here to say which is the best solution in this case.

Hopefully someone else has more knowledge here :)

[faragher]

I have high-level knowledge of the process from my game dev days, but even writing an engine, I didn't implement a system. It was lower priority than core development, which . . . is where we are now.

Please consider this the high-level overview and not a technical specification. None of the pseudocode or JSON has been checked, it's simply there as a broad reference.

At its simplest, you need to replace every single UI string with a token. So:

print("Hello World")

becomes

print(MyTranslator("Hello World"))

MyTranslator is a custom function. It contains all the logic needed to translate a string. It is complicated. I'm going to skip over it to the end. But basically it looks up the token "Hello World" and replaces it with the localized string. If, for example, it was simply stored in a nested dictionary, you could have:

UIStrings["en-US"]["Hello World"] = "Hello World"
UIStrings["de-DE"]["Hello World"] = "Tag Welt" # I don't speak German

So, that's the theory. Now we start descending into details. How do you store it? Ideally you want it compacted into Python, but you can also have it load at runtime which allows dynamically loaded translations, field-upgradeable systems, and a whole new attack vector for malicious code. But at some point you need a human-readable input for the translators. So you'd likely have a file similar to this:

{
  "Hello World": [
    {
      "locale": "en-US",
      "localizedString": "Hello World"
    },
    {
      "locale": "de-DE",
      "localizedString": "Tag Velt"
    },
    {
      "locale": "ru-BY",
      "localizedString": "Привет  мир!"
    }
  ]
}

I have not learned German or Russian since the last code block.

As you can see, from the translator's point of view, it's pretty straightforward, but if 90s video game translation has taught us anything, each would need additional context in the form of comments that were, ideally, only included in the working script, not the shipped product.

And now . . . the pain.

The function requires considerable logic. First, it needs to determine the device's locale, then it needs to check if that locale has been overridden in configuration. Then it needs to load the localization, check if the localization exists, and fall back if it doesn't. And what does it fall back to? If there's a half complete Russian-Belarus translation, do the empty strings default to Russian-Russia or English-United States? Seems obvious, but most people would default to falling back to en-US (or en-UK) depending on the original language. And on that note, should en-US and en-UK map to the same locale, or should the minor spelling differences and regional differences be ignored as the languages are mutually intelligible?

Then there's technical limitations. These are all left to right languages. A right to left language needs more support, hopefully provided by Unicode and Python, but it needs testing and possible position changes in the code based on the order. Are the glyph sizes the same? How much space do we have to display the string? Is this part of the localization data or hardwired in the program? I promise you want it in the localization. Then you want sanitization in case of accidental or malicious bad code. In this example, you could just print the key if the JSON is corrupt, but what happens when you localize something like "Level"? I am using my level to level the floor on the third level of this building. Each concept needs a special localization, not just every word which means every single UI element needs to be checked.

Then do all the same for images.

Here's how it's changed based just on those few issues:

{
  "Fields": {
    "Hello World": [
      {
        "localize": "True",
        "comment": "A statement showing the base program is working",
        "maxlength": "12",
        "rtl_ok": "True"
      }
    ]
  },
  "Localization": {
    "Hello World": [
      {
        "locale": "en-US",
        "fallback": "en-UK",
        "localizedString": "Hello World"
      }
    ]
  }
}

And all of this needs to be looked up at runtime for every single string. And either you load it all into a cache at runtime, or it does the logic every single frame for every single string.

So, what we need is a very detailed specification, a functioning implementation of the handler function, a bunch of testing both for performance and edge cases (especially as we're working with files across multiple platforms), and finally a team of translators, which is apparently the easy part (everyone who's offered, thank you, I sure can't do it).

These are not insurmountable issues, they are simply major, major amounts of effort both conceptually and practically. I'm willing to look into the technical side of things, but as we've all seen, I can be very detailed but very slow. I have not managed to get Buildozer to work to build Sideband, and I don't know where to start for RTL or any of the other edge cases.

If others are interested, I'm more than happy to take point on a specification, but this isn't a job for a single person.

[faragher]

Further information:

I still can't get a build environment set up, (I can complain for hours about the reasons, but they're not useful) but I've done some Kivy digging, as well as some basic thoughts on localization.

First, things look pretty good. Here's an example from Sideband:

                    dm_items = [
                        {
                            "viewclass": "OneLineListItem",
                            "text": "Converse",
                            "height": dp(40),
                            "on_release": gen_conv(context_dest, item)
                        },
                        {
                            "viewclass": "OneLineListItem",
                            "text": "Copy address",
                            "height": dp(40),
                            "on_release": gen_copy_addr(context_dest, item)
                        },
                        {
                            "text": "Delete Announce",
                            "viewclass": "OneLineListItem",
                            "height": dp(40),
                            "on_release": gen_del(context_dest, item)
                        }
                    ]

By converting each of the "text" lines to a token (example: announce_converse) we can call from a dictionary (example: "text":Localization["announce_converse"]) and thus change the strings to whatever we need. If we load and process at program start, it shouldn't add too much to boot time, and be reasonably efficient in runtime. Since these widgets are also generated relatively infrequently, it should be handled well. I think I'm beginning to ramble. Bottom line: absolutely doable with a lookup table.

Generating this table is as above. I won't rehash that, there's no new information.

But, there's some bleeding edge Kivy (Jan 2024) that adds some RTL and other localization support. One of the examples is as follows: kivy/kivy#7704

        return Label(text=text,
                     font_direction="rtl",
                     font_script_name="Arab",
                     font_name="data/Amiri-Regular.ttf" #Arabic supporting font
                     )

If this functions in the widgets as expected, then the "font_direction" directive should solve . . . well . . . the font direction issue. However, it specifies font direction. There's no such thing as mixed language, and it may actually appear backwards in the JSON localization files.