Skip to content

Commit

Permalink
Merge branch 'fix/tokenize-docs' into fix/kanji-iteration-character
Browse files Browse the repository at this point in the history
  • Loading branch information
DJTB committed Sep 28, 2023
2 parents e6e203b + 7e9d999 commit dda897b
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions src/tokenize.js
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,10 @@ export function getType(input, compact = false) {
* tokenize('truly 私は悲しい', { compact: true })
* // ['truly ', '私は悲しい']
*
* tokenize('5romaji here...!?人々漢字ひらがな4カタ カナ「SHIO」。!')
* tokenize('5romaji here...!?人々漢字ひらがなカタ カナ4「SHIO」。!')
* // [ '5', 'romaji', ' ', 'here', '...!?', '人々漢字', 'ひらがな', 'カタ', ' ', 'カナ', '4', '「', 'SHIO', '」。!']
*
* tokenize('5romaji here...!?人々漢字ひらがな4カタ カナ「SHIO」。!', { compact: true })
* tokenize('5romaji here...!?人々漢字ひらがなカタ カナ4「SHIO」。!', { compact: true })
* // [ '5', 'romaji here', '...!?', '人々漢字ひらがなカタ カナ', '4「', 'SHIO', '」。!']
*
* tokenize('5romaji here...!?人々漢字ひらがなカタ カナ4「SHIO」。! لنذهب', { detailed: true })
Expand Down

0 comments on commit dda897b

Please sign in to comment.