Break the language barrier of TFT Color UI

[jmz52]

Description

Break the language barrier of 255 character-fonts inherited from u8glib and embrace unicode support for TFT Color UI.
2 new fonts NotoSans with antialiasing and lightweight Unifont
8 character groups to support all available Marlin translations

• Latin_Extended_A
• Cyrillic
• Greek
• Katakana (with few extra hieroglyphs)
• Korean (limited support)
• Vietnamese (limited support)
• Simplified Chinese (limited support)
• Traditional Chinese (limited support)

Requirements

A lot of flash - fonts with antialiasing are flash hungry.
373 glyphs needed for language_zh_CN.h (Simplified Chinese) use 57Kb for 480x320 UI.

Related Issues

#20268
#21235

[jmz52]

Detailed functionality description can be found in comment below.

To whoever is going to review this: please let me know if you have an elegant way to check for LCD_LANGUAGE settings.
Mine is rather messy - https://github.com/jmz52/Marlin/blob/tft-color-ui-unicode/Marlin/src/lcd/tft/tft_string.h

Character sets showcase:
Latin Extended A / Greek / Katakana / Korean
Cyrillic / Vietnamese / Simplified Chinese / Traditional Chinese

320x240 with NotoSans Medium 14px

320x240 with Unifont 10px

480x320 with NotoSans 19px

[ellensp]

Going to need to add #ifdef blocks to filter this new code out for people not using platformio source filters, such as Arduino IDE users

[quiret]

This looks really good, @jmz52! I appreciate your work a lot.

Do you have any estimates on how this new feature will impact the size of compiled firmware binaries?

[jmz52]

Unicode support and limitations

TFT Color UI is upgraded to support unicode characters instead of uint8_t characters inherited from u8g.
Unicode support is limited to "Basic Multilingual Plane" (0x0000-0xffff).
This limitation saves Flash and RAM by using uint16_t characters instead of uint32_t.

Fonts

Support for u8g fonts is dropped because of font_t header's restrictions - uint8_t variables are used for first and last symbols' code. u8g fonts can be converted to new unifont_t format by replacing font's header.

Helvetica14 (320x240 and 480x272) and Helvetica18 (480x320 and 1024x600) were converted to new font format.
Helvetica18 is now called Helvetica19 because height of capital 'A' is 19px.
Both Helvetica fonts have characters from "Basic Latin" (0x0000-0x007f) and "Latin-1 Supplement" (0x0080-0x00ff) blocks only and can't be used for languages that requires symbols from other blocks.

Optional font anti aliasing. Implemented by using 2 bits per pixel in glyphs' data. AA makes converted TTF/OTF fonts look better at the cost of increased firmware size. Non-AA fonts use 1 bit per pixel.

Two new fonts

• NotoSans (OFL 1.1) with AA is 3 different sizes - 14px, 16px and 19px.
• Unifont (OFL 1.1/GPL 2+) without AA in native (10px) and 2x upscale (20px) sizes.

New TFT_FONT parameter in Configuration.h can be used to select desired font.

Updated UI

Updated UI code for 320xx240, 480x272 and 480x320 resolutions to make them look better with new fonts and get rid of flickering caused by elements overlapping in 480x272 UI.
Minor adjustments to 1024x600 UI to make room for Unifont 20px characters.

'Extra' characters sets

8 'extra' characters sets to support all available LCD_LANGUAGE options
Languages that use characters beyond "Basic Latin" and "Latin-1 Supplement" blocks are grouped by characters used

• Latin Extended-A (full set of 128 characters from Latin Extended-A block 0x0100 - 0x017f) - Czech, Croatian, Polish, Slovak, Turkish

• Cyrillic (74 characters from Cyrillic block 0x400 - 0x4ff) - Bulgarian, Russian, Ukrainian.
  These 74 characters include all characters used in these 3 languages.

• Greek (63 characters from Greek and Coptic block 0x0370 - 0x03ff) - Greek (Greece) and Greek (Cyprus).
  Characters include 24 letters Αα - Ωω, small final sigma ς and 7 vowels with tonos Άά Έέ Ήή Ίί Όό Ύύ Ώώ.
  Not sure if this is a complete set though.

• Katakana (full set of 96 characters from Katakana block 0x30a0 - 0x30ff and 7 extra hieroglyphs from CJK Unified Ideograph 0x4e00 - 0x9fff) - Japanese.
  7 extra hieroglyphs 仮 名 定 片 示 義 表 are needed to display all characters used in language_jp_kana.h file.

• Korean (110 symbols from Hangul Syllables block 0xac00 - 0xd7ff) - Korean (South Korea).
  The only symbols included are symbols used in language_ko_KR.h file.

• Vietnamese (107 letters and diacritical marks from Latin Extended-A 0x0100 - 0x017f, Latin Extended-B 0x0180 - 0x024f, Latin Extended Additional 0x1e00 - 0x1eff and Combining Diacritical Marks 0x0300 - 0x036f blocks) - Vietnamese.
  I was lead to believe that this is a complete set (along with Basic Latin and Latin-1 Supplement) but one can never be too sure.

• Simplified Chinese (373 symbols) - Chinese (Simplified).
  Symbols are gathered from language_zh_CN.h file.

• Traditional Chinese (307 symbols) - Chinese (Traditional).
  Symbols are gathered from language_zh_TW.h file.

• All other languages have no characters outsize of 0x0020-0x00ff range in their language.h files and should be fine with the base character set.

These sets of symbols covers 100% of symbols used in corresponding language.h files, so text lines defined in these files will be correctly displayed on TFT screen.
The only only exception here are 5 Combining Diacritical Marks used in Vietnamese language. These marks requires special logic to be processed as part of preceding letter and are not displayed well without it. Therefore Combining Diacritical Marks are intentionally ignored and not displayed at all.

UTF_FILENAME_SUPPORT

Usage of UTF_FILENAME_SUPPORT is not recommended for Japanese (unless filenames are written with Katakana alphabet), Korean, Chinese (Simplified) and Chinese (Traditional).
Other languages should fine, unless I've missed some characters. Unknown symbols are be displayed as ?, so this will not prevent user from printing from these files.
However one should keep in mind that only one of these extra character sets is loaded, so Marlin with Czech interface will not be able to display filenames written in Greek.

[thinkyhead]

This is looking very nice! I expect most TFT users have some room to spare in Flash so if the size increase is moderate then I'm not too worried. I will merge this for wider testing shortly. I'm particularly interested in finding a way to draw fonts at a larger size for the benefit of the BIQU BX screen. Helvetica 36 ? At least, some numbers and punctuation would be useful to print at the larger size. Of course, the larger screen allows for a lot more information to be displayed on the Info Screen –and with a nicer layout– so that also needs to be expanded upon.

[jmz52]

@thinkyhead, I would rather not invest in Helvetica font.
It was a decent font for initial Color_UI implementation, but its not that useful now.

• Character set is quite limited. Original source code files are from u8glib, so they are by design restricted to 255 characters.
• Freely available .bdf files have maximum font size of 24px and lack the characters requires for eastern languages. And I have not checked if they have Cyrillic characters.
• Conversion from OTF/TTF format requires antialiasing to make a decent looking glyphs (i.e. 2 bits per pixel) and Helvetica is a commercial font, so it can offer no advantages over NotoSans which has all required symbols and is distributed under Open Font License.

Helvetica can still have it uses when firmware size is of an essence and 10px Unifont is too small to be used.

As for larger screens, like 1024x600 of BIQU BX, the bigger fonts is the least problem there.
Icons from 320x240 UI look a bit lonely on 7 inches screen, so both new icons (ideally with a source in vector format for proper scaling) and new UI layout are needed.
All in all this requires a lot more that just font update, so I've intentionally left 1024x600 UI with no improvements in this pull request.

NotoSans and Unifont .cpp files are auto-generated from OTF/TTF files using FreeType2 library and this can done for any other font and fontsize needed.
Speaking of which - it there a repository to store supplementary tools for Marlin where this font converter can be hosted?

[EvilGremlin]

Nope, all in buildroot/share/scripts.
Good idea to add pio target to regenerate fonts/icons. I made some generation scripts, you can expand on that https://github.com/EvilGremlin/3DPrintingIcons https://github.com/EvilGremlin/Marlin/blob/newcolor_ui/Marlin/src/lcd/tft_newcolor/convert.py