feat: use emoji-picker-element-data, make shortcodes optional (#75)

* fix: use emoji-picker-element-data

* fix: working

* fix: fix tests

* fix: more work

* test: tests

* test: more tests

* fix: make shortcodes optional

* docs: fixup

* fix: actually install

* docs: make shortcodes optional

* fix: add deprecation notice to trimEmojiData

* fix: fix trimEmojiData

* docs: update readme

* docs: update readme

* docs: tweak docs and error messages

README.md

@@ -60,7 +60,10 @@ A lightweight emoji picker, distributed as a web component.
     + [Within a Svelte project](#within-a-svelte-project)
   * [Data and offline](#data-and-offline)
     + [Data source and JSON format](#data-source-and-json-format)
-    + [Trimming the emojibase data](#trimming-the-emojibase-data)
+    + [Shortcodes](#shortcodes)
+    + [Cache performance](#cache-performance)
+    + [emojibase-data compatibility (deprecated)](#emojibase-data-compatibility-deprecated)
+    + [Trimming the emoji data (deprecated)](#trimming-the-emoji-data-deprecated)
     + [Offline-first](#offline-first)
     + [Environments without IndexedDB](#environments-without-indexeddb)
   * [Design decisions](#design-decisions)
@@ -109,7 +112,7 @@ This will log:
     "annotation": "grinning face",
     "group": 0,
     "order": 1,
-    "shortcodes": [ "gleeful" ],
+    "shortcodes": [ "grinning_face", "grinning" ],
     "tags": [ "face", "grin" ],
     "unicode": "😀",
     "version": 1,
@@ -247,7 +250,7 @@ Name | Type | Default | Description |
 ------ | ------ | ------ | ------ |
 `customCategorySorting` | function | - | Function to sort custom category strings (sorted alphabetically by default)  |
 `customEmoji` | CustomEmoji[] | - | Array of custom emoji |
-`dataSource` | string | "https://cdn.jsdelivr.net/npm/emojibase-data@^5/en/data.json" | URL to fetch the emojibase data from (`data-source` when used as an attribute) |
+`dataSource` | string | "https://cdn.jsdelivr.net/npm/emoji-picker-element-data@^1/en/emojibase/data.json" | URL to fetch the emoji data from (`data-source` when used as an attribute) |
 `i18n` | I18n | - | i18n object (see below for details) |
 `locale` | string | "en" | Locale string |
 `skinToneEmoji` | string | "🖐️" | The emoji to use for the skin tone picker (`skin-tone-emoji` when used as an attribute) |
@@ -382,7 +385,7 @@ same underlying IndexedDB connection and database.
 Name | Type | Default | Description |
 ------ | ------ | ------ | ------ |
 `customEmoji` | CustomEmoji[] | [] | Array of custom emoji  |
-`dataSource` | string | "https://cdn.jsdelivr.net/npm/emojibase-data@^5/en/data.json" | URL to fetch the emojibase data from |
+`dataSource` | string | "https://cdn.jsdelivr.net/npm/emoji-picker-element-data@^1/en/emojibase/data.json" | URL to fetch the emoji data from |
 `locale` | string | "en" | Locale string |
 
 **Returns:** *Database*
@@ -740,21 +743,42 @@ import Picker from 'emoji-picker-element/svelte';
 
 If you'd like to host the emoji JSON yourself, you can do:
 
-    npm install emojibase-data@^5
+    npm install emoji-picker-element-data@^1
 
-Then host `node_modules/emojibase-data/en/data.json` (or other locales) on your web server.
+Then host `node_modules/emoji-picker-element-data/en/emojibase/data.json` (or other JSON files) on your web server.
 
-`emoji-picker-element` requires the _full_ [`emojibase-data`](https://github.com/milesj/emojibase) JSON file, not the "compact" one (i.e. `data.json`, not `compact.json`). Also note that `emojibase-data` v6 is not yet supported (see [#47](https://github.com/nolanlawson/emoji-picker-element/issues/47)).
+### Shortcodes
 
-It's recommended that your server expose an `ETag` header – if so, `emoji-picker-element` can avoid re-downloading the entire JSON file over and over again. Instead, it will do a `HEAD` request and just check the `ETag`.
+There is no standard for shortcodes, so unlike other emoji data, there is some disagreement as to what a "shortcode" actually is.
 
-If the server hosting the JSON file is not the same as the one containing the emoji picker, then the cross-origin server will also need to expose `Access-Control-Allow-Origin: *` and `Access-Control-Allow-Headers: *`. (Note that `jsdelivr` already does this, which is partly why it is the default.)
+`emoji-picker-element-data` is based on `emojibase-data`, which offers several shortcode packs per language. For instance,
+you may choose shortcodes from GitHub, Slack, Discord, or Emojibase (the default). You
+can browse the available data files [on jsdelivr](https://www.jsdelivr.com/package/npm/emoji-picker-element-data) and see
+more details on shortcodes [in the Emojibase docs](https://emojibase.dev/docs/shortcodes).
 
-Unfortunately [Safari does not currently support `Access-Control-Allow-Headers`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Expose-Headers#Browser_compatibility), meaning that the `ETag` header will not be available cross-origin. In that case, `emoji-picker-element` will fall back to the less performant option. If you want to avoid this, host the JSON file on the same server as your web app.
+### Cache performance
 
-### Trimming the emojibase data
+For optimal cache performance, it's recommended that your server expose an `ETag` header. If so, `emoji-picker-element` can avoid re-downloading the entire JSON file over and over again. Instead, it will do a `HEAD` request and just check the `ETag`.
 
-If you are hosting the JSON file yourself and would like it to be as small as possible, then you can use the utility `trimEmojiData` function:
+If the server hosting the JSON file is not the same as the one containing the emoji picker, then the cross-origin server will also need to expose `Access-Control-Allow-Origin: *` and `Access-Control-Allow-Headers: ETag` (or `Access-Control-Allow-Headers: *` ). `jsdelivr` already does this, which is partly why it is the default.
+
+Note that [Safari does not currently support `Access-Control-Allow-Headers: *`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers#Browser_compatibility), but it does support `Access-Control-Allow-Headers: ETag`.
+
+If `emoji-picker-element` cannot use the `ETag` for any reason, it will fall back to the less performant option, doing a full `GET` request on every page load.
+
+### emojibase-data compatibility (deprecated)
+
+_**Deprecated:** in v1.3.0, `emoji-picker-element` switched from [`emojibase-data`](https://github.com/milesj/emojibase) to
+[`emoji-picker-element-data`](https://npmjs.com/package/emoji-picker-element-data) as its default data source. You can still use `emojibase-data`, but only v5 is supported, not v6. Support may be removed in a later release._
+
+When using `emojibase-data`, you must use the _full_ [`emojibase-data`](https://github.com/milesj/emojibase) JSON file, not the "compact" one (i.e. `data.json`, not `compact.json`).
+
+### Trimming the emoji data (deprecated)
+
+_**Deprecated:** in v1.3.0, `emoji-picker-element` switched from [`emojibase-data`](https://github.com/milesj/emojibase) to
+[`emoji-picker-element-data`](https://npmjs.com/package/emoji-picker-element-data) as its default data source. With the new `emoji-picker-element-data`, there is no need to trim the emoji down to size. This function is deprecated and may be removed eventually._
+
+If you are hosting the `emojibase-data` JSON file yourself and would like it to be as small as possible, then you can use the utility `trimEmojiData` function:
 
 ```js
 import trimEmojiData from 'emoji-picker-element/trimEmojiData.js';

package.json

@@ -79,6 +79,7 @@
     "compression": "^1.7.4",
     "conventional-changelog-cli": "^2.1.0",
     "cssnano": "^4.1.10",
+    "emoji-picker-element-data": "^1.0.0",
     "emojibase-data": "^5.1.1",
     "express": "^4.17.1",
     "fake-indexeddb": "^3.1.2",

src/database/constants.js

@@ -18,5 +18,5 @@ export const MODE_READWRITE = 'readwrite'
 export const INDEX_SKIN_UNICODE = 'skinUnicodes'
 export const FIELD_SKIN_UNICODE = 'skinUnicodes'
 
-export const DEFAULT_DATA_SOURCE = 'https://cdn.jsdelivr.net/npm/emojibase-data@^5/en/data.json'
+export const DEFAULT_DATA_SOURCE = 'https://cdn.jsdelivr.net/npm/emoji-picker-element-data@^1/en/emojibase/data.json'
 export const DEFAULT_LOCALE = 'en'

src/database/customEmojiIndex.js

@@ -17,7 +17,7 @@ export function customEmojiIndex (customEmojis) {
   // search()
   //
   const emojiToTokens = emoji => (
-    [...new Set(emoji.shortcodes.map(shortcode => extractTokens(shortcode)).flat())]
+    [...new Set((emoji.shortcodes || []).map(shortcode => extractTokens(shortcode)).flat())]
   )
   const searchTrie = trie(customEmojis, emojiToTokens)
   const searchByExactMatch = _ => searchTrie(_, true)
@@ -41,7 +41,7 @@ export function customEmojiIndex (customEmojis) {
   const nameToEmoji = new Map()
   for (const customEmoji of customEmojis) {
     nameToEmoji.set(customEmoji.name.toLowerCase(), customEmoji)
-    for (const shortcode of customEmoji.shortcodes) {
+    for (const shortcode of (customEmoji.shortcodes || [])) {
       shortcodeToEmoji.set(shortcode.toLowerCase(), customEmoji)
     }
   }

src/database/dataLoading.js

@@ -5,35 +5,35 @@ import { log } from '../shared/log'
 
 export async function checkForUpdates (db, dataSource) {
   // just do a simple HEAD request first to see if the eTags match
-  let emojiBaseData
+  let emojiData
   let eTag = await getETag(dataSource)
   if (!eTag) { // work around lack of ETag/Access-Control-Expose-Headers
     const eTagAndData = await getETagAndData(dataSource)
     eTag = eTagAndData[0]
-    emojiBaseData = eTagAndData[1]
+    emojiData = eTagAndData[1]
     if (!eTag) {
-      eTag = await jsonChecksum(emojiBaseData)
+      eTag = await jsonChecksum(emojiData)
     }
   }
   if (await hasData(db, dataSource, eTag)) {
     log('Database already populated')
   } else {
     log('Database update available')
-    if (!emojiBaseData) {
+    if (!emojiData) {
       const eTagAndData = await getETagAndData(dataSource)
-      emojiBaseData = eTagAndData[1]
+      emojiData = eTagAndData[1]
     }
-    await loadData(db, emojiBaseData, dataSource, eTag)
+    await loadData(db, emojiData, dataSource, eTag)
   }
 }
 
 export async function loadDataForFirstTime (db, dataSource) {
-  let [eTag, emojiBaseData] = await getETagAndData(dataSource)
+  let [eTag, emojiData] = await getETagAndData(dataSource)
   if (!eTag) {
     // Handle lack of support for ETag or Access-Control-Expose-Headers
     // https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Expose-Headers#Browser_compatibility
-    eTag = await jsonChecksum(emojiBaseData)
+    eTag = await jsonChecksum(emojiData)
   }
 
-  await loadData(db, emojiBaseData, dataSource, eTag)
+  await loadData(db, emojiData, dataSource, eTag)
 }

src/database/idbInterface.js

@@ -6,7 +6,7 @@ import {
   STORE_EMOJI, STORE_FAVORITES,
   STORE_KEYVALUE
 } from './constants'
-import { transformEmojiBaseData } from './utils/transformEmojiBaseData'
+import { transformEmojiData } from './utils/transformEmojiData'
 import { mark, stop } from '../shared/marks'
 import { extractTokens } from './utils/extractTokens'
 import { getAllIDB, getAllKeysIDB, getIDB } from './idbUtil'
@@ -22,10 +22,10 @@ export async function hasData (db, url, eTag) {
   return (oldETag === eTag && oldUrl === url)
 }
 
-export async function loadData (db, emojiBaseData, url, eTag) {
+export async function loadData (db, emojiData, url, eTag) {
   mark('loadData')
   try {
-    const transformedData = transformEmojiBaseData(emojiBaseData)
+    const transformedData = transformEmojiData(emojiData)
     await dbPromise(db, [STORE_EMOJI, STORE_KEYVALUE], MODE_READWRITE, ([emojiStore, metaStore]) => {
       let oldETag
       let oldUrl

src/database/utils/ajax.js

@@ -1,5 +1,5 @@
 import { warnETag } from './warnETag'
-import { assertEmojiBaseData } from './assertEmojiBaseData'
+import { assertEmojiData } from './assertEmojiData'
 import { mark, stop } from '../../shared/marks'
 
 function assertStatus (response, dataSource) {
@@ -24,8 +24,8 @@ export async function getETagAndData (dataSource) {
   assertStatus(response, dataSource)
   const eTag = response.headers.get('etag')
   warnETag(eTag)
-  const emojiBaseData = await response.json()
-  assertEmojiBaseData(emojiBaseData)
+  const emojiData = await response.json()
+  assertEmojiData(emojiData)
   stop('getETagAndData')
-  return [eTag, emojiBaseData]
+  return [eTag, emojiData]
 }

src/database/utils/assertCustomEmojis.js

@@ -1,6 +1,5 @@
 const requiredKeys = [
   'name',
-  'shortcodes',
   'url'
 ]
 
@@ -10,6 +9,6 @@ export function assertCustomEmojis (customEmojis) {
     customEmojis.length &&
     (!customEmojis[0] || requiredKeys.some(key => !(key in customEmojis[0])))
   if (!isArray || firstItemIsFaulty) {
-    throw new Error('Expected custom emojis to be in correct format')
+    throw new Error('Custom emojis are in the wrong format')
   }
 }

src/database/utils/assertEmojiData.js

@@ -0,0 +1,11 @@
+import { requiredKeys } from './requiredKeys'
+
+export function assertEmojiData (emojiData) {
+  if (!emojiData ||
+    !Array.isArray(emojiData) ||
+    !emojiData[0] ||
+    (typeof emojiData[0] !== 'object') ||
+    requiredKeys.some(key => (!(key in emojiData[0])))) {
+    throw new Error('Emoji data is in the wrong format')
+  }
+}

src/database/utils/requiredKeys.js

@@ -3,7 +3,6 @@ export const requiredKeys = [
   'emoji',
   'group',
   'order',
-  'shortcodes',
   'tags',
   'version'
 ]

src/database/utils/transformEmojiBaseData.js → src/database/utils/transformEmojiData.js

@@ -2,13 +2,13 @@ import { MIN_SEARCH_TEXT_LENGTH } from '../../shared/constants'
 import { mark, stop } from '../../shared/marks'
 import { extractTokens } from './extractTokens'
 
-// Transform emojibase data for storage in IDB
-export function transformEmojiBaseData (emojiBaseData) {
-  mark('transformEmojiBaseData')
-  const res = emojiBaseData.map(({ annotation, emoticon, group, order, shortcodes, skins, tags, emoji, version }) => {
+// Transform emoji data for storage in IDB
+export function transformEmojiData (emojiData) {
+  mark('transformEmojiData')
+  const res = emojiData.map(({ annotation, emoticon, group, order, shortcodes, skins, tags, emoji, version }) => {
     const tokens = [...new Set(
       [
-        ...shortcodes.map(extractTokens).flat(),
+        ...(shortcodes || []).map(extractTokens).flat(),
         ...tags.map(extractTokens).flat(),
         ...extractTokens(annotation),
         emoticon
@@ -21,7 +21,6 @@ export function transformEmojiBaseData (emojiBaseData) {
       annotation,
       group,
       order,
-      shortcodes,
       tags,
       tokens,
       unicode: emoji,
@@ -30,6 +29,9 @@ export function transformEmojiBaseData (emojiBaseData) {
     if (emoticon) {
       res.emoticon = emoticon
     }
+    if (shortcodes) {
+      res.shortcodes = shortcodes
+    }
     if (skins) {
       res.skinTones = []
       res.skinUnicodes = []
@@ -42,6 +44,6 @@ export function transformEmojiBaseData (emojiBaseData) {
     }
     return res
   })
-  stop('transformEmojiBaseData')
+  stop('transformEmojiData')
   return res
 }

src/picker/components/Picker/Picker.js

@@ -94,7 +94,7 @@ function unicodeWithSkin (emoji, currentSkinTone) {
 
 // eslint-disable-next-line no-unused-vars
 function labelWithSkin (emoji, currentSkinTone) {
-  return uniq([(emoji.name || unicodeWithSkin(emoji, currentSkinTone)), ...emoji.shortcodes]).join(', ')
+  return uniq([(emoji.name || unicodeWithSkin(emoji, currentSkinTone)), ...(emoji.shortcodes || [])]).join(', ')
 }
 
 //

src/picker/groups.js

@@ -1,4 +1,4 @@
-// via https://unpkg.com/browse/emojibase-data@5.0.1/meta/groups.json
+// via https://unpkg.com/browse/emojibase-data@6.0.0/meta/groups.json
 const allGroups = [
   [-1, '✨', 'custom'],
   [0, '😀', 'smileys-emotion'],

src/picker/utils/summarizeEmojisForUI.js

@@ -23,6 +23,6 @@ export function summarizeEmojisForUI (emojis, emojiSupportLevel) {
     category,
     id: unicode || name,
     skins: skins && toSimpleSkinsMap(skins),
-    title: shortcodes.join(', ')
+    title: (shortcodes || []).join(', ')
   }))
 }

src/trimEmojiData.js

@@ -1,14 +1,19 @@
-import { assertEmojiBaseData } from './database/utils/assertEmojiBaseData'
+import { assertEmojiData } from './database/utils/assertEmojiData'
 import { requiredKeys } from './database/utils/requiredKeys'
 
-const optionalKeys = ['skins', 'emoticon']
+const optionalKeys = ['skins', 'emoticon', 'shortcodes']
 const allKeys = [...requiredKeys, ...optionalKeys]
 
 const allSkinsKeys = ['tone', 'emoji', 'version']
 
-export default function trimEmojiData (emojiBaseData) {
-  assertEmojiBaseData(emojiBaseData)
-  return emojiBaseData.map(emoji => {
+export default function trimEmojiData (emojiData) {
+  console.warn('trimEmojiData() is deprecated and may be removed eventually. ' +
+    'If you use emoji-picker-element-data instead of emojibase-data, there is no need for trimEmojiData(). ' +
+    'For details, see: ' +
+    'https://github.com/nolanlawson/emoji-picker-element/blob/master/README.md##trimming-the-emoji-data-deprecated'
+  )
+  assertEmojiData(emojiData)
+  return emojiData.map(emoji => {
     const res = {}
     for (const key of allKeys) {
       if (key in emoji) {

src/types/database.ts

@@ -8,12 +8,12 @@ export default class Database {
    * Note that multiple Databases pointing to the same locale will share the
    * same underlying IndexedDB connection and database.
    *
-   * @param dataSource - URL to fetch the emojibase data from
+   * @param dataSource - URL to fetch the emoji data from
    * @param locale - Locale string
    * @param customEmoji - Array of custom emoji
    */
   constructor({
-                dataSource = 'https://cdn.jsdelivr.net/npm/emojibase-data@^5/en/data.json',
+                dataSource = 'https://cdn.jsdelivr.net/npm/emoji-picker-element-data@^1/en/emojibase/data.json',
                 locale = 'en',
                 customEmoji = []
               }: DatabaseConstructorOptions = {}) {

src/types/picker.ts

@@ -10,15 +10,15 @@ export default class Picker extends HTMLElement {
 
   /**
    *
-   * @param dataSource - URL to fetch the emojibase data from (`data-source` when used as an attribute)
+   * @param dataSource - URL to fetch the emoji data from (`data-source` when used as an attribute)
    * @param locale - Locale string
    * @param i18n - i18n object (see below for details)
    * @param skinToneEmoji - The emoji to use for the skin tone picker (`skin-tone-emoji` when used as an attribute)
    * @param customEmoji - Array of custom emoji
    * @param customCategorySorting - Function to sort custom category strings (sorted alphabetically by default)
    */
   constructor({
-                dataSource = 'https://cdn.jsdelivr.net/npm/emojibase-data@^5/en/data.json',
+                dataSource = 'https://cdn.jsdelivr.net/npm/emoji-picker-element-data@^1/en/emojibase/data.json',
                 locale = 'en',
                 i18n,
                 skinToneEmoji = '🖐️',

src/types/shared.ts

@@ -13,7 +13,7 @@ export interface NativeEmoji {
   group: number
   name: string
   order: number
-  shortcodes: string[]
+  shortcodes?: string[]
   tags?: string[]
   version: number
   unicode: string
@@ -100,7 +100,7 @@ export interface EmojiPickerEventMap {
 
 export interface CustomEmoji {
   name: string
-  shortcodes: string[]
+  shortcodes?: string[]
   url: string
   category?: string
 }