Flag emojis for England, Scotland, and Wales get turned into black flags

[duxovni]

Steps to reproduce

1. Start composing a message
2. Open the emoji picker
3. Go to the very end of the flags section, where 🏴󠁧󠁢󠁥󠁮󠁧󠁿 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🏴󠁧󠁢󠁷󠁬󠁳󠁿 are
4. Click one of those emojis

Alternatively:

1. Start composing a message
2. Paste in a 🏴󠁧󠁢󠁥󠁮󠁧󠁿, 🏴󠁧󠁢󠁳󠁣󠁴󠁿, or 🏴󠁧󠁢󠁷󠁬󠁳󠁿 emoji from the clipboard

Outcome

What did you expect?

The emoji are displayed properly in the message composer and in the timeline.

What happened instead?

All three of these emoji appear as black flags in the message composer and in the timeline after sending.

It still takes 7 backspace keystrokes to delete them in the composer, which suggests that all of the unicode codepoints are still present, but for some reason only the initial U+1F3F4 Black Flag is displayed, and the remaining codepoints are hidden rather than being combined into the flag properly. The emoji also display properly in "View Source".

I've reproduced this behavior on

• develop.element.io in Firefox
• Element Desktop 1.10.11 on NixOS Unstable
• Element Desktop 1.10.12 on Arch Linux

My guess is this is an unrelated issue from #16884:

• This is reproducible across platforms and between browser/desktop, while the trans flag is only broken in Arch Linux's Element Desktop package
• The 🏴󠁧󠁢󠁥󠁮󠁧󠁿 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🏴󠁧󠁢󠁷󠁬󠁳󠁿 emojis display properly in the emoji picker and in View Source, whereas the trans flag in Arch Desktop is broken into its constituent glyphs everywhere.
• As mentioned, these emoji take 7 backspace keystrokes to delete because the joiner characters aren't being combined properly. The broken trans flag still only takes one keystroke to delete, which suggests it's more an issue around fonts/display.

Operating system

No response

Browser information

No response

URL for webapp

No response

Application version

No response

Homeserver

No response

Will you send logs?

No

[simpleAndrew]

Hey there!

Any plans to get this fixed somehow? :)

[Lamby777]

Hey, I don't know what this project is, so this probably isn't very helpful, but it's worth mentioning since nobody so far has brought up Windows. I found this thread from a Google search when I noticed Flagpedia was showing all the emojis as 2-letter character codes except for England, Scotland, and Wales, which showed up as a black flag. Pretty sure this is a Windows thing, since they refuse to add flag emojis.

[Half-Shot]

I suspect this is due to us splitting emoji's in messages into mx_emoji span tags, and doing a very poor job of complex emoji. https://github.com/matrix-org/matrix-react-sdk/blob/7d0c68aa2922ba6e2d0134ab0999698323bb1f27/src/HtmlUtils.tsx#LL54C1-L57C74 seems to find emoji by regex, and in practice it seems to break things like 🏴󠁧󠁢󠁷󠁬󠁳󠁿. Room previews and the emoji picker actually do render this fine, so the flag isn't broken entirely, it's just the way we process it.

[t3chguy]

We can probably use lodash's split which has proper grapheme splitting

[Johennes]

Looks like we are already doing that (https://github.com/matrix-org/matrix-react-sdk/blob/7d0c68aa2922ba6e2d0134ab0999698323bb1f27/src/HtmlUtils.tsx#L474) but lodash itself can't properly handle that emoji.

for (const char of split("🏴󠁧󠁢󠁷󠁬󠁳󠁿", "")) {
    console.log(`char: ${char}`)
    console.log(`codepoint: ${char.codePointAt(0).toString(16)}`);
}

...

char: 🏴 
codepoint: 1f3f4 
char: 󠁧 
codepoint: e0067 
char: 󠁢 
codepoint: e0062 
char: 󠁷 
codepoint: e0077 
char: 󠁬 
codepoint: e006c 
char: 󠁳 
codepoint: e0073 
char: 󠁿 
codepoint: e007f

Those are all the the code points in "🏴󠁧󠁢󠁷󠁬󠁳󠁿", just not split correctly.

[justjanne]

Lodash just defers to the JS engine's own String#split method. This in turn has two ways of splitting, into surrogate pairs (bad) or into codepoints (which causes this specific issue).

[t3chguy]

@justjanne that isn't right, string split breaks all compound emoji - Lodash uses https://github.com/lodash/lodash/blob/master/.internal/unicodeToArray.js#L39

[t3chguy]

See

[justjanne]

Not really, String.split handles compound emoji just fine if you use /(?:)/u as separator. lodash only uses its own unicodeToArray in rare cases.

[t3chguy]

in rare cases.

Like our case, given we don't pass /(?:)/u as the separator. We fall into the https://github.com/lodash/lodash/blob/master/split.js#L35-L37 unicodeToArray bracket in this emoji handling

[t3chguy]

/(?:)/u also does the wrong thing, it splits 🇩🇪 to regional D, E marks separately.

[Johennes]

Looks like https://github.com/beaugunderson/emoji-aware can split 🏴󠁧󠁢󠁷󠁬󠁳󠁿 properly.

Ironically, their README claims that lodash itself should also do this.

Note: Lodash's toArray (as of 4.0.0) and split (as of 4.9.0) functions now correctly split strings that contain emoji; so if that's all you need to do then Lodash is a great fit.

[justjanne]

Well, it doesn't split 🇩🇪 in my browser, but I can understand why we use lodash's split now (though it's not ideal that we rely on the specific behavior for a specific separator in lodash's split, never been a fan of that)

[t3chguy]

@justjanne we do document the call sites saying that we're relying on lodash's grapheme handling

[t3chguy]

Went for grapheme-splitter instead as its already a transitive dependency

yarn why v1.22.19
[1/4] 🤔  Why do we have the module "grapheme-splitter"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
=> Found "grapheme-splitter@1.0.4"
info Reasons this module exists
   - "@typescript-eslint#eslint-plugin" depends on it
   - Hoisted from "@typescript-eslint#eslint-plugin#grapheme-splitter"
   - Hoisted from "eslint#grapheme-splitter"
info Disk size without dependencies: "252KB"
info Disk size with unique dependencies: "252KB"
info Disk size with transitive dependencies: "252KB"
info Number of shared dependencies: 0
✨  Done in 0.44s.

[Johennes]

Ah, great! 👍