Replace emojify.js with openmoji set #9182
Comments
sapk
added
type/proposal
|
Not bad, but we ned to write own code - openmoji is no drop in place solution it is only a colection of emojis |
|
@6543 Yes, that why I distinguish the two roles of emojify. I wanted to let the first point up to discussion as I don't know if there is a lib that do it. Otherwise, I would plan on forking emojify based on openmoji set + capacity to add specific emoji like the gitea one at "runtime". |
|
I think a custom implementation is the way to go, backed by a minimal name-to-unicode map like https://github.com/muan/emojilib. Most modern browser support unicode emoji so we could get away with just supporting unicode (+custom graphics). If image fallbacks are still desirable, maybe https://github.com/iamcal/js-emoji. |
|
On a sidenote, https://bundlephobia.com/result?p=emojilib@2.4.0 I guess we'd need something more trimmed down with just name to unicode maps without the extra fluff. |
|
I don't think we can use openmoji as it is licensed under cc-by-sa 4.0 that is compatible only with GPLv3 if I understand it correctly :( |
|
@lafriks They mention that it just need a mention somewhere https://github.com/hfg-gmuend/openmoji/blob/master/README.md#attribution-requirements |
|
I don't have any opinions about which emoji library to use, but regarding how to implement it, I'd advise against parsing the emoji on the server (when comments are submitted or when the comments are rendered), i.e. I'm in favor of on-the-fly emoji detection that the user can control and nullify (e.g. using backspace). My reasons:
|
|
Another question we have to ask is whether we want to store emoji as unicode in the database. I think GitHub does it that way and it would make data more portable at the expense of requiring the user to have Unicode support in the DB which may not be given on older MySQL installations. |
|
@silverwind I would prevere the : |
|
We could support both if we used the openmoji font (or any other emoji font) and replace in ui :keyword: with unicode. The :keyword: format is also used in commit message generally. This will also assure backward compat. This could also be implemented in the markdown rendering. |
|
Sorry I linked wrong issue |
|
Just a note: there's also https://twemoji.twitter.com/ which I find much prettier than openmoji. The current emoji set is definitely not very attractive IMO. Would be great to upgrade to a nice new set eventually. |
|
I would actually prefer Google Noto Color Emoji - https://emojipedia.org/google/ https://github.com/googlefonts/noto-emoji |
|
We could maybe use (or develop if not existing) a lib that replace |
|
@sapk I already proposed to use something like https://github.com/muan/emojilib to convert tokens to unicode. I think we should just use the system-provided emoji font as modern OS come with a emoji font. GitHub for example just uses |
Yes, Linux needs a custom font. |
|
It looks like Ubuntu comes with System fonts are the way to go in my opinion. Loading a custom web font just for emoji would impact performance and I guess users will expected to see the emojis in their platform-native form. |
Then why does every major website provide custom fonts? I think there are many cases for a decent fallback, not just Linux that isn't Ubuntu (which, by the way, isn't a vast majority of Linux users either). |
I'd strongly doubt that. Github for example does not rely on a single web font for emoji or regular text which is ideal for performance and eliminates issues like FOUT. |
|
Yes, GitHub is using PNG images instead. I didn't mean that it has to be a webfont, but that you don't want to rely on native emoji fonts only. |
They use Unicode like they should. PNG is only a fallback used on legacy Platforms (I think Windows XP and earlier): <g-emoji alias="+1" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f44d.png" class="emoji mr-1">👍</g-emoji>I'm thinking we just need a config option like |
|
thinking about a implementing a function and register it to the router ...
EDIt: o right - this only solve reaction emojis ... but in generel what do you think of an go solution as plugin to goldmark ...? |
If it can be done server-side, that would be ideal. No flashes of wrong content and best performance. I guess you only need a module to map tokens to unicode codepoints. |
|
and some css -> https://gitea.com/gitea/go-sdk/issues/255#issuecomment-106334 |
|
@silverwind but we cant change all emojis to utf8? -> 🚀 , :gitea:, ... |
|
Yeah, custom ones would need a image. Also have to keep in mind that there are places outside of Markdown that have them, so the replacement would need to run at a higher level, maybe as some sort of post-processing step after template rendering. |
I'm on the very latest Arch Linux release with modern GNOME 3, which also has a native emoji font. I'm getting PNG images on GitHub and basically every other major website. And I also find them nicer to look at than my system's own font. Edit: custom emojis are also a great use case for providing the fallback. Check out e.g. the custom emoji feature in Mastodon to see what I mean. Would be awesome to be able to configure your own additional emojis on every Gitea instance. |
|
I think GitHub does User-Agent sniffing for the decision on whether they serve unicode emoji or fallback images. Try changing to a recent Windows/Mac UA and see if you get Unicode. A major benefit of unicode is that one can copy-paste the text with emojis included (sometimes, they are important to the actual discussion). |
I do not have any other machine with me (while traveling) that would run those operating systems and I also wouldn't buy a Windows license just to test if GitHub delivers emojis in a different way to Windows users. |
|
I just checked what Twitter does, and they're using SVG images with the unicode emoji as Example: This tweet contains emoji 🤯 with image https://abs-0.twimg.com/emoji/v2/svg/1f92f.svg Edit: for inline emojis (as opposed to comment reactions e.g.), GitHub uses a Twemoji webfont. |
|
@silverwind was talking about changing your |
|
Ah, stupid me. Didn't have coffee yet. :) |
|
The alt attribute also permit screen reader to understand the emojis. |
|
Facebook uses PNG images with no <span class="_47e3 _5mfr" title="unsure emoticon">
<img class="img" role="presentation" src="https://static.xx.fbcdn.net/images/emoji.php/v9/t83/1.5/30/1f615.png" alt="" width="30" height="30">
<span aria-hidden="true" class="_7oe">:/</span>
</span>Edit: they also use an actual unicode fallback for other emojis, so they're not all the same: <span style="height: 32px; width: 32px; font-size: 32px;
background-image: url(https://static.xx.fbcdn.net/images/emoji.php/v9/tf9/1.5/32/2764.png);"
class="_6qdm">
❤️
</span> |
That could be solved by a wrapper tag I still think we can do Unicode without fallbacks (except for custom images). Checking http://caniemoji.com/, the only relevant browser targets lost when not serving fallbacks are:
|
|
From what I know screen reader understand unicode in alt attr (even if it make weird definition/translation) and that why twitter put it there. |
Sure, but at least ship an emoji webfont then, like GitHub does. |
GitHub does not ship any webfonts. Where are you getting this from? |
|
Ah, sorry. I hadn't inspected it well enough. Mozilla is shipping the Twemoji font with Firefox apparently. |
- remove emojify and replace it with custom name-to-unicode replacement - move tribute code to own file as it depends on emoji data This depends on simplemap.json that emojilib publishes which is a 40kB JSON file containing name-to-unicode mappings. It supports 1571 emoji compared to the previous 881 of emojify. Fixes: go-gitea#9182 Fixes: go-gitea#8974 Fixes: go-gitea#8953
This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea. This works in a few ways: First it adds emoji parsing support into gitea itself. This allows us to * Render emojis from valid alias (:smile:) * Detect unicode emojis and let us put them in their own class with proper aria-labels and styling * Easily allow for custom "emoji" * Support all emoji rendering and features without javascript * Uses plain unicode and lets the system render in appropriate emoji font * Doesn't leave us relying on external sources for updates/fixes/features That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also) For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method. The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released. I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens. I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others. Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary. Fixes: go-gitea#9182 Fixes: go-gitea#8974 Fixes: go-gitea#8953 Fixes: go-gitea#6628 Fixes: go-gitea#5130
This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea. This works in a few ways: First it adds emoji parsing support into gitea itself. This allows us to * Render emojis from valid alias (:smile:) * Detect unicode emojis and let us put them in their own class with proper aria-labels and styling * Easily allow for custom "emoji" * Support all emoji rendering and features without javascript * Uses plain unicode and lets the system render in appropriate emoji font * Doesn't leave us relying on external sources for updates/fixes/features That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also) For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method. The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released. I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens. I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others. Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary. Fixes: go-gitea#9182 Fixes: go-gitea#8974 Fixes: go-gitea#8953 Fixes: go-gitea#6628 Fixes: go-gitea#5130
This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea. This works in a few ways: First it adds emoji parsing support into gitea itself. This allows us to * Render emojis from valid alias (:smile:) * Detect unicode emojis and let us put them in their own class with proper aria-labels and styling * Easily allow for custom "emoji" * Support all emoji rendering and features without javascript * Uses plain unicode and lets the system render in appropriate emoji font * Doesn't leave us relying on external sources for updates/fixes/features That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also) For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method. The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released. I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens. I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others. Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary. Fixes: go-gitea#9182 Fixes: go-gitea#8974 Fixes: go-gitea#8953 Fixes: go-gitea#6628 Fixes: go-gitea#5130
* Support unicode emojis and remove emojify.js This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea. This works in a few ways: First it adds emoji parsing support into gitea itself. This allows us to * Render emojis from valid alias (:smile:) * Detect unicode emojis and let us put them in their own class with proper aria-labels and styling * Easily allow for custom "emoji" * Support all emoji rendering and features without javascript * Uses plain unicode and lets the system render in appropriate emoji font * Doesn't leave us relying on external sources for updates/fixes/features That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also) For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method. The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released. I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens. I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others. Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary. Fixes: #9182 Fixes: #8974 Fixes: #8953 Fixes: #6628 Fixes: #5130 * add new shared function emojiHTML * don't increase emoji size in issue title * Update templates/repo/issue/view_content/add_reaction.tmpl Co-Authored-By: 6543 <6543@obermui.de> * Support for emoji rendering in various templates * Render code and review comments as they should be * Better way to handle mail subjects * insert unicode from tribute selection * Add template helper for plain text when needed * Use existing replace function I forgot about * Don't include emoji greater than Unicode Version 12 Only include emoji and aliases in JSON * Update build/generate-emoji.go * Tweak regex slightly to really match everything including random invisible characters. Run tests for every emoji we have * final updates * code review * code review * hard code gitea custom emoji to match previous behavior * Update .eslintrc Co-Authored-By: silverwind <me@silverwind.io> * disable preempt Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Support unicode emojis and remove emojify.js This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea. This works in a few ways: First it adds emoji parsing support into gitea itself. This allows us to * Render emojis from valid alias (:smile:) * Detect unicode emojis and let us put them in their own class with proper aria-labels and styling * Easily allow for custom "emoji" * Support all emoji rendering and features without javascript * Uses plain unicode and lets the system render in appropriate emoji font * Doesn't leave us relying on external sources for updates/fixes/features That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also) For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method. The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released. I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens. I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others. Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary. Fixes: go-gitea#9182 Fixes: go-gitea#8974 Fixes: go-gitea#8953 Fixes: go-gitea#6628 Fixes: go-gitea#5130 * add new shared function emojiHTML * don't increase emoji size in issue title * Update templates/repo/issue/view_content/add_reaction.tmpl Co-Authored-By: 6543 <6543@obermui.de> * Support for emoji rendering in various templates * Render code and review comments as they should be * Better way to handle mail subjects * insert unicode from tribute selection * Add template helper for plain text when needed * Use existing replace function I forgot about * Don't include emoji greater than Unicode Version 12 Only include emoji and aliases in JSON * Update build/generate-emoji.go * Tweak regex slightly to really match everything including random invisible characters. Run tests for every emoji we have * final updates * code review * code review * hard code gitea custom emoji to match previous behavior * Update .eslintrc Co-Authored-By: silverwind <me@silverwind.io> * disable preempt Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
emojify.js is not maintained anymore.
At some point we would need to switch to another lib to manage emoji (or maintain a fork).
The lib emojify.js does two think :
I suggest to move to openmoji for the emoji set.
It seems to be a large choice with good quality : https://openmoji.org/library/
The text was updated successfully, but these errors were encountered: