Issues with UTF-8 texts between windows and *NIX

[kuhnchris]

Godot version

3.3.3.stable

System information

Windows 10, GLES3, Linux (Arch) GLES3, Android

Issue description

When trying to work with a RichTextEdit everything works fine, the special [font] tag allows me to add smileys inline. All of this works dandy fine in Linux and Android, but on Windows we have the issue that the smiley copied from here: https://en.wikipedia.org/wiki/Emoticons_(Unicode_block)
are not visible in the editor anymore. Even worse, when trying top "copy" the unicode smileys from there, we have the following in the RichTextEdit, and no smiley available on display.

Same on Linux, but with working rendering:

I assume this is a Unicode/Windows/Locale/... issue - but are there any pointers or hints on how to work around that?

Steps to reproduce

• Clone https://github.com/kuhnchris/godot-sandbox-whatsapp-but-not/
• Open project in godot editor
• See (or do not see) smiley in 2nd box. (after the bee movie block)

Minimal reproduction project

https://github.com/kuhnchris/godot-sandbox-whatsapp-but-not/

[bruvzg]

This is known issue.

Godot 3.x use wchar_t as a string char type (on Windows it's 16-bit, on other platforms 32-bit), and do not have surrogate pair support. It can't display characters with charcode > 0xffff on Windows. Fixing it would be breaking change and unlikely to happen for 3.x.

Godot 4.x use 32-bit char32_t for strings on all platforms and do not have this issue.

[akien-mga]

This is not the first time this comes up, is there a place we could document this in 3.x so that's it's known why it doesn't work, and that it won't be fixed in 3.x?

[bruvzg]

Or we can add partial support for surrogates to the dynamic font, input won't work entirely correctly, but displaying emojies should be possible without breaking changes (and it's probably the most common use for it, at least all previous issues were about emojies).

draw_char/get_char_size do receive next char for kerning anyway, we can use it to combine surrogates.

[kuhnchris]

This is not the first time this comes up, is there a place we could document this in 3.x so that's it's known why it doesn't work, and that it won't be fixed in 3.x?

I checked the RichTextEdit doc first, I assumed if there is anything regarding that it would be there.
Or a separate "Unicode" section?

[Calinou]

This was originally reported in #23093.

I agree that adding support for surrogates in 3.x would be a welcome change.

[akien-mga]

Fixed by #54625.