Skip to content

Commit 0ebd665

Browse files
committed
Fix gemma3_text tokenizer in mapping (#36793)
1 parent 46350f5 commit 0ebd665

File tree

1 file changed

+7
-0
lines changed

1 file changed

+7
-0
lines changed

src/transformers/models/auto/tokenization_auto.py

+7
Original file line numberDiff line numberDiff line change
@@ -222,6 +222,13 @@
222222
"GemmaTokenizerFast" if is_tokenizers_available() else None,
223223
),
224224
),
225+
(
226+
"gemma3_text",
227+
(
228+
"GemmaTokenizer" if is_sentencepiece_available() else None,
229+
"GemmaTokenizerFast" if is_tokenizers_available() else None,
230+
),
231+
),
225232
("git", ("BertTokenizer", "BertTokenizerFast" if is_tokenizers_available() else None)),
226233
("glm", (None, "PreTrainedTokenizerFast" if is_tokenizers_available() else None)),
227234
("gpt-sw3", ("GPTSw3Tokenizer" if is_sentencepiece_available() else None, None)),

0 commit comments

Comments
 (0)