Tokenizer's Interfaces Cleanup #7001
Conversation
|
@michaelgsharp I appreciate it if you could review the changes. I have removed a couple of APIs you introduced earlier and provided a workaround for their usage. Thank you! |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #7001 +/- ##
==========================================
- Coverage 68.81% 68.80% -0.02%
==========================================
Files 1258 1258
Lines 250477 250652 +175
Branches 25576 25602 +26
==========================================
+ Hits 172377 172468 +91
- Misses 71473 71553 +80
- Partials 6627 6631 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
|
| internal List<Token> TokenizeWithCache(string sequence) | ||
| { | ||
| List<Token> tokens = new(word.SymbolsCount); | ||
| Word word; | ||
| if (Cache is not null) | ||
| { | ||
| if (Cache.TryGet(sequence, out word)) | ||
| { | ||
| return WordToTokens(ref word); | ||
| } | ||
|
|
||
| foreach (Token token in word.GetIterator(VocabReverse)) | ||
| word = MergeWord(sequence); | ||
| Cache.Set(sequence, word); | ||
| } | ||
| else | ||
| { | ||
| tokens.Add(token); | ||
| word = MergeWord(sequence); | ||
| } | ||
|
|
||
| return tokens; | ||
| return WordToTokens(ref word); |
There was a problem hiding this comment.
Can this whole method just be:
List<Token> result = new();
TokenizeToIdsWithCache(sequence, result);
return result;?
There was a problem hiding this comment.
No, TokenizeToIdsWithCache(sequence, result); fill only the Ids and removing the overhead for filling the whole tokens. Note the difference in the implementation between TokenizeToIdsWithCache and TokenizeWithCache. The first is calling WordToIds while the second is calling WordToTokens
| /// <param name="sequence">The sequence to split.</param> | ||
| /// <param name="isSpecialToken">Indicate if the token is a special token.</param> | ||
| /// <param name="accumulatedIds">The list of accumulated tokenized Ids.</param> | ||
| public override void TokenizeToIds(string sequence, bool isSpecialToken, IList<int> accumulatedIds) => TokenizeToIds(sequence, accumulatedIds); |
There was a problem hiding this comment.
When do we use the word Tokenize vs the word Encode?
There was a problem hiding this comment.
We use the Encode name at the Tokenizer class level to process the entire input text. At the level of the tokenizer's models, we use the Tokenize name, which operates on smaller, pre-tokenized text units.
This update encompasses the following:
Tokenizer.GetEncodedIdsCountAPI, essential for supporting crucial scenarios and implemented it in all supported tokenizers.EncodeToIdsandGetEncodedIdsCounthas been customized for other tokenizer models likeBpeandEnglishRoberta. This adaptation aims to enhance the performance of these APIs specifically when invoked from those respective tokenizers.