Console width length off-by-one on characters that are 2 cells in width?

[starfishpatkhoo]

Hmm, I am not sure where this issue seems to be rooted.

1. Take the Sample clinkprompt and activate it
2. Modify rightfilter()

function p:rightfilter()
    local char_symbol = "⚡"  -- 0x26a1
    -- console.cellcount(char_symbol) == 2
    return char_symbol..os.date("%a %H:%M", os.time())
end

3. Launch terminal, but then run cmd again (nested)
4. Notice the timestamp getting pushed over the end of the console line into the next line

If I use Windows Terminal, it only happens after I nest cmd (run cmd inside of cmd), but this happens on a pure cmd.exe without Windows Terminal as well.

I admit I did not have a chance to investigate whether this happens on rightfilter() only, or also on filter().. Nor what other characters might be affected. I believe it is those characters that a larger than usual... Thing is, console.cellcount() reports the correct answer for this character, so I really don't know where the issue is... :(

[chrisant996]

U+26A1 is known to be rendered inconsistently in Windows consoles.

There's no way for Clink to accurately predict how it will be rendered.

Don't use that character.

Use U+F0E7 with nerd fonts v3 (or use U+E007 with nerd fonts v2). And this is why flexprompt uses those instead of U+26A1.

[starfishpatkhoo]

Ah, okie.. Well noted.. I shall keep that in mind... I chose a different character in the end, cos the bolt was starting to get too distracting..

[chrisant996]

Tracked it down: U+26A1 is a "color emoji" Unicode codepoint. On Win11, conhost and Windows Terminal both draw it as a monochrome single-width glyph but give it two character cells, instead of drawing it as a double-width color emoji.

But when cmd.exe is nested, then it can't detect Windows Terminal, and so it thinks color emoji aren't supported, and it assumes 1 cell for color emoji codepoints. But at least on Win11, the codepoints get rendered as double-width (even when they draw only a single-width glyph -- because the cell width needs to stay consistent even if the glyphs end up with a space after them).

I'll test Win8.1 and Win10 and Win11 to see which versions draw "color emoji" codepoints as double-width, and which "color emoji" get different treatment. I wonder if in Win8.1 and/or Win10 conhost draws them as single-width glyphs in a single-width cell (because I tested all this, so how did the behavior get broken?). It's also possible that Win11 changed the behavior, or it's even possible that a certain version of Windows Terminal changed the behavior.

Unfortunately, it can also depend on the specific selected font, but there isn't a reliable way for programs to query the current console font.

It's difficult and time consuming to try to keep up with the ever-changing matrix of which codepoints get rendered in what way on what versions of Windows and/or Windows Terminal. It may not be realistic to try to support more than "latest build number" of each version of Windows (8.1, 10, 11) and "latest version" of Windows Terminal. I'll see what I can do.

[chrisant996]

Addressed by dc31308.

The width for U+26A1 (and other Unicode color emoji codepoints) should generally be predicted accurately now, for the latest Win8.1/Win10/Win11 versions, and for conhost and Windows Terminal and ConEmu. I can't say for sure whether the predictions will hold true in other terminal programs, as they have freedom to render things according to their own choices.

Caveat: There are 122 color emoji that Windows Terminal draws as being two cells wide, but only move the cursor by 1 cell. I haven't yet compensated for those, because it looks like a bug in Windows Terminal. I'll open an issue in the Windows Terminal repo for that.

[chrisant996]

Caveat: There are 122 color emoji that Windows Terminal draws as being two cells wide, but only move the cursor by 1 cell. I haven't yet compensated for those, because it looks like a bug in Windows Terminal. I'll open an issue in the Windows Terminal repo for that.

Oh, of course: my wcwidth-verifier tool is printing only the color emoji codepoints by themselves. It does not add U+FE0F to fully qualify the color emoji. Due to historical issues in the earliest terminal programs that supported color emoji, the unqualified forms have to be treated as 1 cell wide, and the fully-qualified forms are treated as 2 cells wide.

Clink already handles the fully-qualified forms and measures them as 2 cells wide. But I haven't yet forced the unqualified forms to be treated as 1 cell wide. I'll make a table of those and add exceptions for them into Clink's width measurements.

That should get things even closer to accurate for most content. But some emoji sequences are too complex for Clink to realistically handle, and the Unicode spec and DirectX/DirectDraw implementation is constantly evolving to keep up with the Unicode spec, so it's a moving target anyway.

This comment in the Windows Terminal repo has a lot of related background info about why this is intentional behavior.

[chrisant996]

Clink v1.7.4 will include improvements to width measurements for many color emoji codepoints (commit ae5ea42). And skin tone selectors should work in many color emojis, as long as then don't use other joiners or selectors (and by "work" I mean the width measurement should generally be accurate; the actual rendering is handled by the terminal itself).

[chrisant996]

And commit c382dba adds further emoji width improvements:

• emoji sequences using the zero width joiner character are recognized (for example 🏳️ ZWJ 🌈 is how 🏳️‍🌈 is produced, ❤️ ZWJ 🔥 is how ❤️‍🔥 is produced, 🤷 ZWJ ♀️ is how 🤷‍♀️ is produced, etc).
• skin tone variant selectors work even in emoji sequences with ZWJ.
• country flag sequences are recognized (although Windows Terminal doesn't yet render them as actual flags).