A tokenizer is in charge of preparing the inputs for a model.
The tokenizer can tokenize Chinese-English bilingual in Linux.
This project mainly solves some Chinese character encoding problems.
Requirements
- Boost
-
Notifications
You must be signed in to change notification settings - Fork 19
ericperfect/libtorch_tokenizer
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
BERT Tokenizer in C++
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published