Unicode/Latin2/ISO_8859_2 characters like É Á Ő Ű etc

[nagyrobi]

Trying this example on Lanbon L8: https://fvanroie.github.io/hasp-docs/#example-2/
Noticed that special characters in word "Átvíztűrőtükörfúrógép" are skipped if I send them over via MQTT.
I found the description about built-in fonts bit that only specifies how to set the size, not the font itself. Any hint on how to select it correctly? The dropdown box is empty in the web interface.

[nagyrobi]

Uploading HMI UbuntuCondensed-Regular 12 (ISO_8859_2).zi from your other repository unsuccessful.
Renamed file to something without spaces and that resulted in a bootloop after reboot, only NETWIZE logo is flashing. :(

EDIT: I'll try with recompiling.

[nagyrobi]

It looks like robotocondensed_regular_XX_nokern.c is completely missing the 0x0A0-0x0FF Latin-1 Supplement and 0x100-0x17F Latin Extended-A characters.

But in all sizes it includes the FontAwesome5 symbols.

:(

[nagyrobi]

Font handling could be optimized a bit:

• pre-create .c arrays with various font selections and sizes and make them selectable through user_config_override.h or platformio-override.ini
• don't include FontAwesome in the same files with the normal characters and in all sizes, instead, create separate FontAwesome font files of multiple sizes. Typically on a screen, icons look nice if they are all the same size, including them in multiple sizes is a waste of space imho. One can still recompile with different icon sizes if they want to. Or perhaps leave out FontAwesome completely and offer it as uploadable .zi files. Maybe someone would like MDI instead...
• also including all characters of all languages can be space-consuming so maybe it should be considered creating font files containing only the characters specific to languages. Like translations are done. One could select the preferred language font files through user_config_override.h or platformio-override.ini. Ask for contributions from users with suggested language files with their characters. I can contribute two.
• have a separate, optional font file containing only numbers, dots etc., for displaying numeric values bigger (like size 34 or 44). Suggestion: 0x020,0x021,0x02B-0x02E,0x030-0x03A,0x03F.

Based on the above I was able to add support for my language's diacritics, include an extrabig number set and free up 121,8kB of flash space.

[fvanroie]

Much like LVGL, the icons and characters are purposefully put into the same font file to allow you to mix icons and text seamlessly. Also, the character set is the same in all font sizes to allow you to easily scale up or down by just changing the point size. No need to know which letter is in which font, either the character is supported or it is not...

For the default built-in fonts, simplicity is paramount. Having icons/characters dependent on font and size will add confusing and complexity. I can imagine the default character set to be dependent on the firmware language. When you install the HU version "Átvíztűrőtükörfúrógép" are also available in the default built-in fonts, which seems reasonable and manageable.

Some middle ground will have to be found between added characters, icons and font sizes without overly complicating the number of available firmware options.

As always, users are free to compile their own customized versions of the firmware including the fonts.
The default firmware is a general purpose solution, that should fit most use-cases.

[nagyrobi]

Agree, but how will this all be possible when we already have 95% of the flash full? My approach was to try to save as much space as possible because we'ver ran into issues going over 100%...

[fvanroie]

Flash is not the main issue, we can re-partition it to increase the firmware size, while reducing the user portion of the filesystem. The constraint is RAM because all characters are loaded into RAM at boot whether you use them or not...

As with all micro-controllers, you need to balance features and performance with the resources available.
Trying to find an acceptable least common denominator is what is the difficult part.

[nagyrobi]

Here is my suggestion for a basic character set, which removes usually unused (at least less likely to be used on hasp-lvgl displays) characters from Basic Latin-1, and adds some useful from Latin-1 Supplement (like degree symbol, +/- sign, micro, quarters:
0x020,0x021,0x025,0x028-0x03A,0x03C-0x05B,0x05D,0x05F,0x061-0x07A,0x07C,0x0B0,0x0B1,0x0B5,0x0B7,0x0BC-0x0BE

New font files in .c array format can be generated with an online tool: https://lvgl.io/tools/fontconverter
Tips:

• use your desired font in ttf or woff format
• Name: type with all small caps the desired name and size (e.g. roboto_condensed_12)
• Size: 12, Bpp: 2 bit-per-pixel
• Range: a comma-separated list of the characters desired in the font file, like above. Character codes list (note the code numbers show without 0x)

[nagyrobi]

Submitted a PR with fomt-selector from user_config_override.h: #83

[fvanroie]

While you can certainly do this to fit your individual needs, the goal of the default firmware is to get a device set up easily and usable.
Some users want more icons and ascii-only, while other users might want Cyrillic characters and no icons...

Additional fonts are on the roadmap, to be added on-the-fly to the filesystem instead of compiling in the firmware.
.zi fonts are currently broken, and this needs to be investigated and fixed. Lvgl also has a way to dynamically load fonts that can help fix this problem.

Changing fonts using #defines seems hard to manage going forward, there are so many different language scripts.
I'm aiming to have a default minimal font that supports the basic setup and examples, with configurable extra fonts on the filesystem.

[nagyrobi]

I have thought so, that's why it's implemented so that the original methods aren't affected. I'd keep this until .zi fonts get a fix, and until dynamic loading gets implemented...

I didn't know that it's known that .zi are broken per se, all I've noticed that uploading and setting a zi font creates a bootloop which can be recovered by flashing only. I thought maybe I'm doing somehing wrong.

[fvanroie]

I have found this overview of alphabets. It might be a useful exercise to determine which letters are common in multiple alphabets and languages. I can run a script to generate the .c files for different fonts and compare the file sizes.

Common letters can be added to the default font, but some icons probably need to go... it's a balancing act, but I'm open for suggestions.

[nagyrobi]

PRs submitted

[nagyrobi]

If you change in lang.h to #include "hu_HU.h"

Also the WiFi setup first screen loos nice.

[nagyrobi]

I see lots of strings on the web interface which could be translated but are not (yet) in the language file. I think it's not a problem to maintain new strings appearing.

[fvanroie]

Will.submit localized versions of these for Hu, Ro.

Thanks! Those files have been merged now.

I see lots of strings which could be translated but are not (yet) in the language file

Indeed, this is a work in progress and I add new ones directly and old ones are moved when I encounter them

[nagyrobi]

Would you like me to create a configurable option for choosing the language in user_config_override.h? That would automatically select the correct font files containing the appropriate characters and language files. On the same principle as my previous PR for selecting the fonts.

[fvanroie]

I generate the fonts with a script and I'm adding the ranges to it to see what is the difference in file size.

[fvanroie]

Adding most latin international characters to the base font will add about 120+ characters and increase the font size by 50%. Consequently a lot of custom icons would have to be removed again. A quick ask on the Discord channel shows that not all users want localized characters and some prefer icons over text for touch displays... so there is a range of opinions.

So, the idea is now to make the default character set dependent on the firmware language installed.

[nagyrobi]

This is great.

[fvanroie]

I'm thinking of compiling a list like this one for the characters used in different languages. The CSV can be easily opened in a spreadsheet to be edited or filtered.

Then that list can be used to generate the fonts automatically e.g. font/hu/robotocondensed_regular_12_nokern.c
If possible, I want to keep the variable names of the fonts equal accross all languages, so lv_conf_v7.h doesn't need a lot of #ifdef's, but would just refer to robotocondensed_regular_xx_nokern regardless of the language selected.

This will need some clever src_filter or keep the files in a separante repo to include in a custom environment... not sure yet :)

[nagyrobi]

I suggest an extra column before the first language, called something like "present in default" or "present in all". To be able to show which symbols should be present or not, regardless of the language.

Or should we mark these in the same columns as the language? For example every language should have the non-breaking space, and the degree symbol...

[fvanroie]

I suggest to just add all languages to that line. That is easiest way for a script to read the file without having to check another column...

[nagyrobi]

And the standard Latin-1 characters like ABCDEF...?

[nagyrobi]

I suggest to just add all languages to that line. That is easiest way for a script to read the file without having to check another column...

PR done

[fvanroie]

Ascii 0x20 - 0x7E will be added automatically as these are also used by LVGL.
The list is for the extended characters on top of ASCII.

[fvanroie]

This is tied to the new material design icons and those new fonts will be available with other character-sets as well.