[Maintenance] Bump minimum tokenizers to 0.15 (#3856)

ludwig-ai · Jan 4, 2024 · 9621757 · 9621757
1 parent 2b52be6
commit 9621757
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/requirements.txt b/requirements.txt
@@ -14,7 +14,7 @@ pydantic<2.0
 transformers>=4.36.2
 tifffile
 imagecodecs
-tokenizers>=0.13.3
+tokenizers>=0.15
 spacy>=2.3
 PyYAML>=3.12,<6.0.1,!=5.4.* #Exlude PyYAML 5.4.* due to incompatibility with awscli
 absl-py