Get proper width reporting for utf8 Thai script

[pepa65]

Describe the feature

The .len_utf8() function on strings gives the right number of utf-8 glyphs, but when tone or other marks or certain vowels are used that are displayed under or over a consonant, there is no way in V to find out the displayed length. There is no accounting for zero-width characters.

The .utf8_str_visible_length() function gives the same result as the .len_utf8 function, and the .east_asian.display_width() function (in encoding.utf8.east_asian) also returns the same (incorrect) result.

For example, the word ผู้ is composed of 3 utf8 glyphs, but is displayed in 1 character position.

Use Case

When calculating the displayed length of words or phrases in the Thai language.

Proposed Solution

Account for zero-width iso glyphs.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

Version used

V 0.3.3 c16549b

Environment details (OS name and version, etc.)

Linux Mint 21.1

[pepa65]

This program is another example of not taking into account zero-width characters:

names := ['1234567890', 'ตาล', 'René', 'ยอห์น', 'ปีเตอร์']
for name in names {
  println('ชื่อ: ${name:10s}')
}

Output:

ชื่อ: 1234567890
ชื่อ:        ตาล
ชื่อ:       René
ชื่อ:      ยอห์น
ชื่อ:    ปีเตอร์

This is V 0.4.1 fdabd27 now. It should be considered a bug, and not a discussion, there is nothing really to discuss I think. Taking into account zero-width characters is not particularly hard, many languages and little programs have done it.

Note that the é is also 2 runes, but somehow 0xcc81 handled correctly.

[peppergrayxyz]

V doesn't handle grapheme clusters. I create a feature request for this: #22117

[medvednikov]

Thanks for reporting.