Partially Addresses #2616. Support combining sequences that don't normalize

[BDisp]

Partially addresses (but does not fix) #2616

Pull Request checklist:

• I've named my PR in the form of "Fixes #issue. Terse description."
• My code follows the style guidelines of Terminal.Gui - if you use Visual Studio, hit CTRL-K-D to automatically reformat your files before committing.
• My code follows the Terminal.Gui library design guidelines
• I ran dotnet test before commit
• I have made corresponding changes to the API documentation (using /// style comments)
• My changes generate no new warnings
• I have checked my code and corrected any poor grammar or misspellings
• I conducted basic QA to assure all features are working

[tig]

PR to this PR incoming with suggested fixes to Unit Tests...

[tig]

Please merge #2934 and see my comment.

I don't think this fix does what you think it does.

[BDisp]

@tig I already finished this.

[tig]

@tig I already finished this.

I don't think your test in TabView is working. Here's what it should look like:

Here's what I see:

Another test (in Unicode scenario):

sb.Append ('e');
sb.Append ('\u0301');
sb.Append ('\u0301');
testlabel = new Label ($"Should be an e with two accents: {sb}") { X = 20, Y = Pos.Y (label), Width = Dim.Percent (50), CanFocus = true, HotKeySpecifier = new Rune ('&') };
Win.Add (testlabel);

Like this:

[tig]

I still don't think this is working. See my comments.

[tig]

Perhaps we shouldn't try normalizing at all? Like this:

Re: NetDriver - I suspect Console.Write can't handle combining marks.

[BDisp]

Perhaps we shouldn't try normalizing at all? Like this:

There is no problem on the AddRune because on the UpdateScreen I can get the correct string in the output variable,

Re: NetDriver - I suspect Console.Write can't handle combining marks.

Yes I believe is that.

[tig]

Perhaps we shouldn't try normalizing at all? Like this:

There is no problem on the AddRune because on the UpdateScreen I can get the correct string in the output variable,

Do you have an objection to replacing the current code in this PR with the code I suggested above that removes the calls to normalize? (with modifications to deal with CombiningMarks being a list)

[BDisp]

Do you have an objection to replacing the current code in this PR with the code I suggested above that removes the calls to normalize?

@tig did you already saw what it do? You are replacing the previous p

Do you have an objection to replacing the current code in this PR with the code I suggested above that removes the calls to normalize?

This is worst.

[BDisp]

Absolute chaos.

[BDisp]

@tig sorry but I don't have more head to handle with this. I think it's reasonable for now, attended to the difficult degree you added to this.

[BDisp]

Without normalize WindowsDriver returns as follow. Separated combining marks output as double ´´. See the dealignment at right.

[tig]

I dove in deep this morning and I think I understand how all this should work much better. I composed a message on my home computer but didn't have a chance to finish my thoughts before I had to run. As soon as I get home I'll finish it.

The net is: We can make combining marks work on both NetDriver and WindowsDriver, but the current design of AddRune/AddStr/UpdateScreen will not work. In addition TextFormatter (and others) are doing things in such a way that will not work.

[tig]

This is not sufficient. Many, many, characters cannot be combined with a combining mark, not just ' '. Try, for example [.

WT (and other platforms) have sophisticated algorithms for figuring out whether a combining mark can combine with a base character. I am trying to find more details on these to see what we can do.

You can easily test this in WT and PS with (Caskadia Cove Nerd Font):

In Unicode, not all base characters can be combined with combining characters. Whether a base character can be combined with a combining character depends on the rules defined in the Unicode Standard. The Unicode Consortium, the organization responsible for maintaining the Unicode Standard, specifies these rules.

Here are some general guidelines to determine if a base character can be combined with a combining character:

Character Composition Model: Unicode follows a character composition model, which means that some characters can be composed from a base character and one or more combining characters. This allows for a wide range of characters and diacritics to be represented in text.

Compatibility: Unicode defines compatibility characters and compatibility composition rules to ensure that combining characters work as expected with base characters. If a base character is defined as a "combining character target," it means that it can be combined with one or more combining characters.

Normalization Forms: Unicode defines normalization forms (NFC, NFD, NFKC, NFKD) to handle character composition and decomposition. NFC (Normalization Form C) is the most commonly used form for combining characters, and it ensures that text is represented in a composed form when possible.

Character Properties: Unicode assigns specific properties to each character, including whether it can be used as a base character and whether it can combine with combining characters. You can refer to the Unicode Character Database (UCD) to check the properties of individual characters.

Combining Class: Combining characters are assigned a "combining class" value, which determines their combining behavior. Base characters and combining characters are combined according to their combining class values.

Compatibility Decomposition: Some base characters can be combined using compatibility decomposition mappings, even if they don't have explicit combining characters. These mappings are defined in the Unicode Standard.

To determine whether a specific base character can be combined with a combining character, you can consult the Unicode Standard documentation, specifically the Unicode Character Database (UCD) and the Unicode Technical Reports related to normalization and composition. Additionally, Unicode-aware text processing libraries and programming languages often provide functions or methods for handling character composition and decomposition, making it easier to work with combined characters in text processing.

[BDisp]

@tig I only had to do that because of the CharacterMap scenario using the space to separate the runes to avoid incorrect output. I don't have enough knowledge to do better.

[tig]

Good info:

• Funadmental Flaws inherent in the design of AtlasEngine microsoft/terminal#16132
• [Epic] Text Buffer rewrite  microsoft/terminal#8000

[tig]

@tig try this code on WindowsDriver on the WriteToConsole method. It seems to support true color but I need help.

Happy to. May be a few days though.

I'm not seeing how this is relevant to combining sequences though. Can you explain?

[BDisp]

No it's not. Forget.

[tig]

• Combining Marks have GetColumns() == 0
• TextFormatter.Draw is currently doing var runeWidth = Math.Max (rune.GetColumns (), 1); and thus incrementing the column when a combining mark is encountered.
• AddRune always assumes a combining mark is 0 width and should be combined with the preceding Cell (unless Col == 0).
• AtlasEngine (Console.Write("a\u0301\u0301")) DOES render multiple CMs correctly.

However, it ALSO moves the cursor to the right for each CM. Thus Console.Write("a\u0301\u0301\u0328x") results in:

Note, this is the same thing you see with echo "[au{0301}u{0301}x]":

If I SetCursorPostion() to the row/col to just to the right of the a, WT 'forgets' the CM.

In otherwords, internally AtlasEngine is holding each CM as a used column.

AtlasEngine also, treats individual CMs (Console.Write("\u0301")) as 1 column width glyphs if in the 1st column.

Unless, I'm missing something, until AtlasEngine changes, on Windows, we cannot really support NON-NORMALIZED combining marks in a way that users would expect. Any glyph made up of a base char and n or more combining chars will use base.ColumnWidth() + n columns.

I think for us to support combining marks correctly we need to do the following:

• Always try to normalize. This at least lets the most common combining mark use-case (base + 1 CM) work in many cases.
• Have ConsoleDriver.AddRune treat a combining mark as a 1 column wide glyph. E.g. if rune.IsCombiningMark() == true then ConsoleWidth == 1.
• Have ConsoleDriver.AddStr enumerate the runes in str and add them to ConsoleDriver.Contents such that Cell.CombiningMarks is setup correctly (if str[0] is a CM, special case).
• Drivers' UpdateScreen
  • treat any Cell.Rune.IsCombiningMark() == true as taking 1 column.
  • use Cell.Rune.CombiningMarks().Count > 0 to append the marks to output.

[tig]

Please see my PR to this PR: BDisp#162

I do not think #2616 can be fixed at this time. I think the best we can do is basically ignore CMs that can't be normalized. My PR to your PR does this.

[BDisp]

Please see my PR to this PR: BDisp#162

I do not think #2616 can be fixed at this time. I think the best we can do is basically ignore CMs that can't be normalized. My PR to your PR does this.

Done @tig. Thanks.

[BDisp]

@tig why you write u301 twice here \u0301\u0301\u0328?

[tig]

@tig why you write u301 twice here \u0301\u0301\u0328?

It was just an example of 3 CMs.

[BDisp]

@tig why you write u301 twice here \u0301\u0301\u0328?

It was just an example of 3 CMs.

I see. For have sure it could be normalized. You don't know if there is 3 different combining marks that can normalize on terminal? It would be better.

[tig]

@tig why you write u301 twice here \u0301\u0301\u0328?

It was just an example of 3 CMs.

I see. For have sure it could be normalized. You don't know if there is 3 different combining marks that can normalize on terminal? It would be better.

\u0301\u0301\u0328 will not normalize! It results in

e\u0301\u0301\u0328 will not normalize. It results in:

You can pile on as many \u0301 as you want:

e\u0301 WILL normalize! AtlasEngine does NOT attempt to normalize (note used cell before ]. Also note the if you copy the glyph and paste it you get e\u301 back. If it were normalizing you'd get \u00e9 (Latin Small Letter E With Acute - é U+000e9).