Full Unicode Support #2541

cloudwu · 2019-05-07T09:02:21Z

This PR add full unicode support, See #2538

I defined two types: ImWchar16 and ImWchar32, and the macro ImWchar is ImWchar16 by default. We can change it in imconfig.h .
ImTextCharFromUtf8 replace the code point above 0x10000 to replacement character U+FFFD when we use ImWchar16. So that we can remove some range checks outside.
ImTextCharToUtf8 and ImTextCountUtf8BytesFromChar can manage full unicode set now. btw, There is a bug (for surrogate) in ImTextCountUtf8BytesFromChar before.
GetClipboardTextFn_DefaultImpl and SetClipboardTextFn_DefaultImpl have some issues with surrogate of UTF-16 before, I reimplement them by ::WideCharToMultiByte and ::MultiByteToWideChar.
I introduce a new API ImGuiIO::AddInputCharacterUTF16, it can handle UTF-16 correctly. examples/imgui_impl_win32.cpp assumed that one code point per WM_CHAR message before. It's not correct, because the wParam can be surrogate, we should combine two WM_CHAR message when the wParam is surrogate.
For the ImFontGlyphRangesBuilder, UsedChars may 17x larger then before if we use ImWchar32.

I hope I haven't left something out.

ocornut · 2019-05-07T15:11:00Z

Thank you @cloudwu !
It may take me a while to chew through this as I am a little busy and not very familiar with all of this.

Quick notes:

(0) Do you have realistic expectation as to which ranges you or other people would consider using 0x10000..0x10FFFF ?
Could you give me instructions to test the input of some of those characters? (e.g. U+20628 𠘨) or whatever is easier, so I could later test with multiple back-ends.

(1) I think we should remove the filtering from the back-end/examples code. I have a small local patch over master to change the signature to AddInputCharacter(unsigned int). Then the ImGui function can freely perform filtering (e.g. if < 0x10000 for current code) and the back-end doesn't need to know about it. I can commit that already with my patch if you agree with this (may relate to point (2)).

(2) Do you see any issue with making AddInputCharacter() behave like your AddInputCharacterUTF16() and remove that new function? How do popular back-ends behave in respect to surrogate pairs? GLFW seems to pass the WM_CHAR/UNICHAR data straight to the user. SDL in its WIN_ConvertUTF32toUTF8 function appears to expect characters above 0x10000 passed to WM_CHAR and converts them to 4-byte UTF-8 sequences, which unless I am wrong seems to contradict what you are doing in imgui_impl_win32.cpp. Is SDL wrong, or are those messages affected by MSVC compilation settings?

(3) I haven't tried the patch yet but I suspect maybe ImFileOpen() would be broken as we cast ImWchar to wchar_t?

(4) I guess ImFontGlyphRangesBuilder will overflow when used with ImWchar16.

(5) If your git-fu makes you able to rebase this into two commits, you can separate the Win32 clipboard stuff into a separate commit (and maybe ImFileOpen fix) and i'll cherry-pick the first one faster.

Thanks!

…ume the input is an unicode code point, not UTF-16

cloudwu · 2019-05-08T03:21:54Z

Unicode has 17 planes now ( https://en.wikipedia.org/wiki/Plane_(Unicode) ), 0-FFFF is BMP (plane 0), SIP (plane 2) is rarely uncommon Chinese characters/ Kanji, I have seen some of them in my friends' name. SMP( plane 1) maybe more useful for emoji . For example, U+1F604 (😀) . You can find them here : https://unicode.org/emoji/charts/emoji-list.html .

If you want to test rendering SIP character, you may need find a font including them. I guess you can got one from your windows system. Have a look to your registry with the key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\LanguagePack\SurrogateFallback . In my computer, C:\Windows\fonts\simsunb.ttf is a font for plane 2.

Testing WM_CHAR for surrogate pair is a little complicated. You should know how to input the character or emoji from IME. I make an user-defined word with character in plane 2 in my IME to test.

You can commit your patch in master, I can rebase it.
My opinion is reserving the AddInputCharacter for unicode code point. I think UTF-16's surrogate pair is not a good solution for Unicode. There is a similar issue for surrogate pairs : Convert UTF-16 input to UTF-32 on Windows SFML/SFML#366

I read the document of glfw, the glfwSetCharCallback of glfw treat the input as the native endian UTF-32 . https://www.glfw.org/docs/latest/input_guide.html

Because an unsigned int is 32 bits long on all platforms supported by GLFW, you can treat the code point argument as native endian UTF-32.

I don't know why SDL pass the UTF-32 to WM_CHAR, but MSDN suggest using WM_UNICHAR instead. And UTF-32 has no surrogate pair, so it can be converted to UTF-8 directly.

I fixed it.
I haven't seen the overflow issue, I'll check it again. I think if the user use ImWchar16 , they would not use the range over 0x10000. btw, Maybe you can manage ranges internal by plane. Each plane is a 16bit
space. Most of users use only one plane (BMP), some of us may want to use SMP for emoji or SIP for some rarely name.
It's done. I think 2cba94a cf4c041 3bedb30 can be merged first.

cloudwu · 2019-05-08T03:56:07Z

I don't know why AppVeyor build failed, it's strange that it can't find MultiByteToWideChar but OpenClipboard is ok. I use mingw and haven't visual studio environment to test .

nicolasnoble · 2019-05-08T04:58:47Z

Try including stringapiset.h after windows.h.

cloudwu · 2019-05-08T06:44:35Z

Try including stringapiset.h after windows.h.

It doesn't work.

nicolasnoble · 2019-05-08T07:00:28Z

Oh. Reading my own code, I just realized these functions are defined behind #if defined UNICODE || defined _UNICODE

cloudwu · 2019-05-09T01:56:58Z

Oh. Reading my own code, I just realized these functions are defined behind #if defined UNICODE || defined _UNICODE

It seems that the imgui project has already defined UNICODE.

nicolasnoble · 2019-05-09T02:50:05Z

It seems that the imgui project has already defined UNICODE.

What? Where? The only reference in the code I could find was in some build_win32.bat files that aren't used by the appveyor script.

cloudwu · 2019-05-09T03:00:36Z

I haven't a visual studio , but I guess the IDE will define UNICODE if charset is set to UNICODE. https://github.com/ocornut/imgui/blob/master/examples/example_win32_directx11/example_win32_directx11.vcxproj#L29

nicolasnoble · 2019-05-09T03:53:08Z

You are correct, but only for the projects that do define this. The OpenGL projects that the solution is also trying to build don't have this setting.

But after double checking appveyor, it's the DX project that's failing, not the OpenGL one... So... I dunno.

nicolasnoble · 2019-05-09T03:54:41Z

@ocornut you think you could increase the verbosity of the msbuild command on appveyor so we can see what's going on here with the command-line arguments?

ocornut · 2019-05-09T07:35:25Z

It doesn’t matter what each examples/ project uses, we should get imgui to compile/link properly on every combination of user’s settings?

nicolasnoble · 2019-05-09T13:29:36Z

Yes, but right now, I am confused as to why it currently doesn't build exactly.

…Character(unsigned int c). Examples/Backends: Don't filter characters under 0x10000 before calling io.AddInputCharacter(), the filtering is done in io.AddInputCharacter() itself. This is in prevision for fuller Unicode support. (#2538, #2541)

+ sponsors update.

Integrated power saving mode in Allegro example. Fixed the cursor-blinking issue. It is now working as expected, i.e. only setting a frame rate requirement of 6fps when visibly blinking. Also refactored a bit such that ImGui uses a boolean flag instead of sharing the FrameRateRequirements object with user code. Renamed FrameRateRequirements -> UserFrameRateRequirements. Just so it's more explicit. Minor formatting fixes [noci] Implemented support for event waiting timeout in Win32. Simplified the implementation. This also addresses a design issue regarding how the application requests a specific frame rate. Get rid of the minimum frame rate. Implemented logic to always render at least 3 frames. Plus a bit of renaming/refactoring. Implemented the 3-frame update logic for the SDL example. Added support for 3-frame updates to Allegro example + refactoring of SDL/OpenGL3 example + refactoring and bug fix of Win32/DX11 example Added suppor for 3-frame updates to Glfw example. Rebased imstb_rectpack on stb_rect_pack v1.00. SliderScalar: Improved assert when using U32 or U64 types with a large v_max value. (ocornut#2765) + misc minor stuff. Demo: PlotLine example displays the average value. (ocornut#2759) + extra comments Added a mechanism to compact/free the larger allocations of unused windows (buffers are compacted when a window is unused for 60 seconds, as per io.ConfigWindowsMemoryCompactTimer = 60.0f). Note that memory usage has never been reported as a problem, so this is merely a touch of overzealous luxury. (ocornut#2636) Disable with ConfigWindowsMemoryCompactTimer < 0.0f (ocornut#2636) TabBar: improved shrinking for large number of tabs to avoid leaving extraneous space on the right side. Individuals tabs are given integer-rounded width and remainder is spread between tabs left-to-right. TabBar: feed desired width (sum of unclipped tabs width) into layout system to allow for auto-resize. (ocornut#2768) Before 1.71 tab bars fed the sum of current width which created feedback loops in certain situations. Amend f95c77e. DragInt, DragFloat, DragScalar: Using (v_min > v_max) allows locking any edit to the value. ColorEdit: Disable Hue edit when Saturation==0 instead of letting Hue values jump around. Fixed missing IMGUI_API for IsMouseDragPastThreshold(). Renamed SetMaxTimeBeforeNewFrame -> SetMaxWaitBeforeNextFrame Compute the proper time to flip the cursor, instead of using 6fps. Due to the 3-frame logic, this makes the effective cursor frame rate go from 15fps to 5fps. Refactored the event waiting implementation in SDL. Moved the waiting part into the common implementation file. This makes the platform+renderer binding simpler and not deviating much from the existing one, which should make merging/integration easier, especially for people maintaining their own copies. Renamed variable to match the method name. Refactored Win32 waiting code into common platform file + fix typo in SDL example Fixed build Avoid wasting frames by adding a small margin for the cursor. Fixed regression Win32 example: don't render 3 frames on timeouts + some refactoring Refactored+fixed the Allegro5 implementation Refactored+fixed Glfw example Refactored+fixed SDL example Refactoring (renaming) Some final cleanup/refactoring; make the diff better Refactored: no need to conditionally (not) poll after wait. Keep waiting when the window is hidden (or minimized). Implemented for Allegro, Glfw and SDL. (does not seem to work with Allegro on Linux) Implemented blocking when minimized/hidden for Win32 Minor formatting/doc changes. Knocked off the 3 TODO items implemented by this PR. Added missing Glfw callback (mouse pos) This is just to record the event. The mouse pos is still handled as before. Small optimization: added shortcut test when feature disabled This avoids paying the library/syscall cost when not needed. Backends: OpenGL3: Tweaked initialization code allow application calling ImGui_ImplOpenGL3_CreateFontsTexture() before ImGui_ImplOpenGL3_NewFrame() if for some reason they wanted. Fix DragScalar for unsigned types (ocornut#2780) decreasing the value was broken on arm64 Nav, Scrolling: Added support for Home/End key. (ocornut#787) Columns: Separator: Fixed a bug where non-visible separators within columns would alter the next row position differently than visible ones. Fixed rounding issues also leading to change of ScrollMax depending on visible items (in particular negative coordinate would be rounded differently) Fix signed types warning in pasteboard handler (ocornut#2786) Examples: SDL/GLFW + OpenGL3: Fixes for Makefile (ocornut#2774) - append CXXFLAGS instead of overwriting them - add glad.c build rule BeginTabItem: Fixed case where right-most tab would create an extraneous draw calls (probably related to other tab fitting code in 1.73 wip) Remove trailing spaces (grep for ' \r?$' in visual studio) Internal: Offset STB_TEXTURE_K_ defines to remove that change from ocornut#2541 + sponsors update. Font: implement a way to draw narrow ellipsis without relying on hardcoded 1 pixel dots. (ocornut#2775) This changeset implements several pieces of the puzzle that add up to a narrow ellipsis rendering. `ImFontConfig` and `ImFont` received `ImWchar EllipsisCodePoint = -1;` field. User may configure `ImFontConfig::EllipsisCodePoint` a unicode codepoint that will be used for rendering narrow ellipsis. Not setting this field will automatically detect a suitable character or fall back to rendering 3 dots with minimal spacing between them. Autodetection prefers codepoint 0x2026 (narrow ellipsis) and falls back to 0x0085 (NEXT LINE) when missing. Wikipedia indicates that codepoint 0x0085 was used as ellipsis in some older windows fonts. So does default Dear ImGui font. When user is merging fonts - first configured and present ellipsis codepoint will be used, ellipsis characters from subsequently merged fonts will be ignored. Rendering a narrow ellipsis is surprisingly not straightforward task. There are cases when ellipsis is bigger than the last visible character therefore `RenderTextEllipsis()` has to hide last two characters. In a subset of those cases ellipsis is as big as last visible character + space before it. `RenderTextEllipsis()` tries to work around this case by taking free space between glyph edges into account. Code responsible for this functionality is within `if (text_end_ellipsis != text_end_full) { ... }`. There are cases when font does not have ellipsis character defined. In this case RenderTextEllipsis() falls back to rendering ellipsis as 3 dots, but with reduced spacing between them. 1 pixel space is used in all cases. This results in a somewhat wider ellipsis, but avoids issues where spaces between dots are uneven (visible in larger/monospace fonts) or squish dots way too much (visible in default font where dot is essentially a pixel). This fallback method obsoleted `RenderPixelEllipsis()` and this function was removed. Note that fallback ellipsis will always be somewhat wider than it could be, however it will fit in visually into every font used unlike what `RenderPixelEllipsis()` produced. Font: Narrow ellipsis: various minor stylistic tweaks (ocornut#2775) Font: Narrow ellipsis: once we know an ellipsis is going to be drawn, we can claim the space between pos_max.x and ellipsis_max.x which gives us enough extra space to not requires the further (and otherwise valid) optimizations. Gets us vastly simplified code, yay. (ocornut#2775) Style: Allow style.WindowMenuButtonPosition to be set to ImGuiDir_None to hide the collapse button. (ocornut#2634, ocornut#2639) + Fix ocornut#2775 ImDrawListSplitter: fixed an issue merging channels if the last submitted draw command used a different texture. (ocornut#2506) Fixed unused static function warning for some compilers. (ocornut#2793) TreeNode: Added ImGuiTreeNodeFlags_SpanAvailWidth and ImGuiTreeNodeFlags_SpanFullWidth flags (ocornut#2451, ocornut#2438, ocornut#1897) Added demo bits. Warning fix. ColorPicker / ColorEdit: restore Hue when zeroing Saturation. (ocornut#2722, ocornut#2770) Issue is fixed by storing last active color picker color and last hue value when active color picker takes rgb as input. Then if current color picker color matches last active color - hue value will be restored. IDs are not used because ColorEdit4() and ColorWidget4() may call each other in hard-to-predict ways and they both push their own IDs on to the stack. We need hue restoration to happen in entire stack of these widgets if topmost widget used hue restoration. Since these widgets operate on exact same color value - color was chosen as a factor deciding which widgets should restore hue. ColorPicker / ColorEdit: restore Hue when zeroing Saturation. (ocornut#2722, ocornut#2770) - changelog, fixed uninitialized variables, tweaks, renaming. Fixed mouse event forwarding in macos example (ocornut#2710, ocornut#1961) Readme, Wiki: Image loading examples.

ocornut · 2019-10-09T15:29:21Z

Closing this (reason: moved to #2815)
Thank you!

fix build for WideCharToMultiByte [3181ff1e] Full Unicode Support [6c9e73ac] Fix ImTextCountUtf8BytesFromChar and ImTextCharToUtf8, these APIs assume the input is an unicode code point, not UTF-16 [ba85665b] Add AddInputCharacterUTF16 for windows backend to handle WM_CHAR [fafdcaf0] Use Windows API to convert UTF-16 for ImFileOpen [dc7d5925] Use windows API to convert UTF-16 for clipboard

- Make ImWchar32 unsigned. - Fix Win32 version of ImFileOpen by including windows.h sooner. - Make ImGuiIO::AddInputCharacterUTF16() more robust by disallowing illegal surrogate pairs. - Allow pushing higher plane codepoints through ImGuiIO::AddInputCharacter(). - Minor cleaning up in the high-plane Unicode support. - Fix Clang -Wunreachable-code warning

…2815)

…ems unnecessary?). (#2541, #2815)

ocornut · 2020-03-11T12:26:44Z

Replying to:
#2541 (comment)

Including <stringapiset.h> breaks Windows SDK 7 (VS2010) and it seemed unnecessary in the test I did, so I removed this include for now.

CI logs:
https://github.com/ocornut/imgui/actions/runs/53633110

…litate C-binding. (#2538, #2541, #2815)

#2541, #2815)

…rnut#2538) fix build for WideCharToMultiByte [3181ff1e] Full Unicode Support [6c9e73ac] Fix ImTextCountUtf8BytesFromChar and ImTextCharToUtf8, these APIs assume the input is an unicode code point, not UTF-16 [ba85665b] Add AddInputCharacterUTF16 for windows backend to handle WM_CHAR [fafdcaf0] Use Windows API to convert UTF-16 for ImFileOpen [dc7d5925] Use windows API to convert UTF-16 for clipboard

…nut#2815) - Make ImWchar32 unsigned. - Fix Win32 version of ImFileOpen by including windows.h sooner. - Make ImGuiIO::AddInputCharacterUTF16() more robust by disallowing illegal surrogate pairs. - Allow pushing higher plane codepoints through ImGuiIO::AddInputCharacter(). - Minor cleaning up in the high-plane Unicode support. - Fix Clang -Wunreachable-code warning

…cornut#2538, ocornut#2815)

…ems unnecessary?). (ocornut#2541, ocornut#2815)

…litate C-binding. (ocornut#2538, ocornut#2541, ocornut#2815)

…nut#2538, ocornut#2541, ocornut#2815)

Amend ef13d95 + c8ea0a0 (#2541, #2538, #2815)

cloudwu mentioned this pull request May 7, 2019

unicode inputChar support for imguiBeginFrame bkaradzic/bgfx#1756

Merged

ocornut added the font/text label May 7, 2019

cloudwu added 5 commits May 8, 2019 10:10

Use windows API to convert UTF-16 for clipboard

2cba94a

Use Windows API to convert UTF-16 for ImFileOpen

cf4c041

Add AddInputCharacterUTF16 for windows backend to handle WM_CHAR

fc2dff0

Fix ImTextCountUtf8BytesFromChar and ImTextCharToUtf8, these APIs ass…

3bedb30

…ume the input is an unicode code point, not UTF-16

Full Unicode Support

a9c741c

cloudwu force-pushed the unicode branch from 42aacb9 to a9c741c Compare May 8, 2019 02:52

fix build for WideCharToMultiByte

01c4d56

cloudwu force-pushed the unicode branch from eaf080d to 01c4d56 Compare May 8, 2019 06:38

ocornut force-pushed the master branch from 1b84280 to aca6ee1 Compare May 11, 2019 09:35

Merge branch 'master' into unicode

44eb8e6

ocornut mentioned this pull request May 21, 2019

ImFontGlyphRangesBuilder has multiple memory issues #2568

Closed

Merge branch 'master' into unicode

f524371

ocornut force-pushed the master branch from c426094 to 54c49b5 Compare July 2, 2019 16:34

ocornut force-pushed the master branch from 51085b5 to baae057 Compare July 23, 2019 20:40

ocornut added a commit that referenced this pull request Sep 17, 2019

Internal: Offset STB_TEXTURE_K_ defines to remove that change from #2541

3f986e7

+ sponsors update.

ocornut added the inputs label Sep 21, 2019

samhocevar mentioned this pull request Sep 29, 2019

Refreshed full Unicode support patch #2815

Closed

ocornut closed this Oct 9, 2019

ocornut added a commit that referenced this pull request Mar 3, 2020

Unicode: Changelog, comments, minimum CI integration. (#2541, #2538, #…

a413324

…2815)

ocornut added a commit that referenced this pull request Mar 11, 2020

Unicode, Windows: Remove stringapiset.h include (breaks vs2010 and se…

4a10af2

…ems unnecessary?). (#2541, #2815)

ocornut added a commit that referenced this pull request Mar 24, 2020

Added IMGUI_USE_WCHAR32 instead of "#define ImWchar ImWchar32" to fac…

670367e

…litate C-binding. (#2538, #2541, #2815)

ocornut added a commit that referenced this pull request Mar 25, 2020

Alter definition of IM_UNICODE_ defines to faclitate C-binding. (#2538,

1d4b5de

#2541, #2815)

sergeyn pushed a commit to sergeyn/imgui that referenced this pull request Mar 30, 2020

Unicode: Changelog, comments, minimum CI integration. (ocornut#2541, o…

76789b3

…cornut#2538, ocornut#2815)

sergeyn pushed a commit to sergeyn/imgui that referenced this pull request Mar 30, 2020

Unicode, Windows: Remove stringapiset.h include (breaks vs2010 and se…

c640186

…ems unnecessary?). (ocornut#2541, ocornut#2815)

sergeyn pushed a commit to sergeyn/imgui that referenced this pull request Mar 30, 2020

Added IMGUI_USE_WCHAR32 instead of "#define ImWchar ImWchar32" to fac…

0bbb684

…litate C-binding. (ocornut#2538, ocornut#2541, ocornut#2815)

sergeyn pushed a commit to sergeyn/imgui that referenced this pull request Mar 30, 2020

Alter definition of IM_UNICODE_ defines to faclitate C-binding. (ocor…

6d8a5ea

…nut#2538, ocornut#2541, ocornut#2815)

ocornut added a commit that referenced this pull request May 20, 2020

IO: AddInputCharacters function ignore 0 input. (#3252)

c8cde28

Amend ef13d95 + c8ea0a0 (#2541, #2538, #2815)

ocornut mentioned this pull request Sep 28, 2022

Russian input in ImGuiInputText #5725

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Full Unicode Support #2541

Full Unicode Support #2541

cloudwu commented May 7, 2019 •

edited

Loading

ocornut commented May 7, 2019

cloudwu commented May 8, 2019 •

edited

Loading

cloudwu commented May 8, 2019 •

edited

Loading

nicolasnoble commented May 8, 2019

cloudwu commented May 8, 2019

nicolasnoble commented May 8, 2019

cloudwu commented May 9, 2019

nicolasnoble commented May 9, 2019

cloudwu commented May 9, 2019

nicolasnoble commented May 9, 2019

nicolasnoble commented May 9, 2019

ocornut commented May 9, 2019

nicolasnoble commented May 9, 2019

ocornut commented Oct 9, 2019

ocornut commented Mar 11, 2020 •

edited

Loading

Full Unicode Support #2541

Full Unicode Support #2541

Conversation

cloudwu commented May 7, 2019 • edited Loading

ocornut commented May 7, 2019

cloudwu commented May 8, 2019 • edited Loading

cloudwu commented May 8, 2019 • edited Loading

nicolasnoble commented May 8, 2019

cloudwu commented May 8, 2019

nicolasnoble commented May 8, 2019

cloudwu commented May 9, 2019

nicolasnoble commented May 9, 2019

cloudwu commented May 9, 2019

nicolasnoble commented May 9, 2019

nicolasnoble commented May 9, 2019

ocornut commented May 9, 2019

nicolasnoble commented May 9, 2019

ocornut commented Oct 9, 2019

ocornut commented Mar 11, 2020 • edited Loading

cloudwu commented May 7, 2019 •

edited

Loading

cloudwu commented May 8, 2019 •

edited

Loading

cloudwu commented May 8, 2019 •

edited

Loading

ocornut commented Mar 11, 2020 •

edited

Loading