An easy way to compatible special characters in mention and hashtag

[smilecc]

By reading the source code, I found an easy way to compatible special characters in mention.

Although I'm not sure, it's strong or not, however, It's working.

How to do (Only Mention)

Talk is cheap, the code:

import * as linkify from 'linkifyjs';
import { DOMAIN } from 'linkifyjs/lib/linkify/core/tokens/text';

const { scanner } = (linkify as any);
scanner.start.on(/[\u2E80-\u9FFF]/, new scanner.State(DOMAIN));

/[\u2E80-\u9FFF]/ is the RegExp of Chinese, Japanese, and Korean character sets, replace them with what you want at will.

Before

After

Why?

This library work through character matching and state transition.

Mention plugin to tell the core how to match and state transfer. When the next state is null, the matching is finished.

But the character sets of this library did not contain what we needed (some special chars), Because this library default just have ASCII chars. so it's always null.

So just add the character set we need, it seems to work well.

[smilecc]

Hashtag Support

Compared to mention, the method of supporting hashtag is HACK. Because S_DOMAIN is not export from scanner.js.

So I had to search the whole state tree.

If you used this code, you don't need to use Only Mention's code.

import * as linkify from 'linkifyjs';
import { DOMAIN } from 'linkifyjs/lib/linkify/core/tokens/text';

const ALPHANUM = '0123456789abcdefghijklmnopqrstuvwxyz';
const APPEND_REG = /[\u2E80-\u9FFF]/;

const scanner = (linkify as any).scanner;
const stateDomain = new scanner.State(DOMAIN);
stateDomain.processed = true;
stateDomain
  .on(ALPHANUM.split(''), stateDomain)
  .on(APPEND_REG, stateDomain);

function deepSet(deep: number, state: any) {
  if (deep > 3) {
    return;
  }

  state.on(APPEND_REG, stateDomain);
  state.processed = true;

  for (const item of state.j) {
    const [keyword, nextState] = item;
    if (!nextState.processed && ALPHANUM.includes(keyword)) {
      deepSet(deep + 1, nextState);
    }
  }

  return state;
}

deepSet(0, scanner.start);

Before

After

[smilecc]

I hope this can help you
@alirezamaleky @timhuebener

[nfrasser]

@smilecc, thanks so much for writing this up!

FYI, internationalized mention and hashtag support is getting added to linkify 3.0 and is now available for preliminary testing in the latest v3.0.0-beta.1. Production release coming soon.

[nfrasser]

Fixed in v3

[toger5]

How is this supposed to work now? I cannot get a tt with a character array or an regex expression to work

[nfrasser]

@toger5 see details here: #167 (comment)