Skip to content

Commit

Permalink
Update openvino_tokenizers (#680)
Browse files Browse the repository at this point in the history
  • Loading branch information
@Wovchena
Wovchena authored Jul 24, 2024
1 parent 03590c5 commit 53945f7
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion thirdparty/openvino_tokenizers
Submodule openvino_tokenizers updated 35 files
+2 −0 .github/workflows/labeler.yml
+17 −6 .github/workflows/linux.yml
+6 −2 .github/workflows/mac.yml
+130 −14 .github/workflows/windows.yml
+1 −0 .gitignore
+12 −3 CMakeLists.txt
+230 −126 README.md
+2 −1 pyproject.toml
+17 −6 python/openvino_tokenizers/cli.py
+11 −0 python/openvino_tokenizers/constants.py
+11 −1 python/openvino_tokenizers/convert_tokenizer.py
+215 −60 python/openvino_tokenizers/hf_parser.py
+7 −19 python/openvino_tokenizers/tiktoken_parser.py
+113 −62 python/openvino_tokenizers/tokenizer_pipeline.py
+52 −7 python/openvino_tokenizers/utils.py
+55 −9 src/CMakeLists.txt
+52 −0 src/byte_fallback.cpp
+36 −0 src/byte_fallback.hpp
+1 −1 src/icu4c.patch
+8 −0 src/icu_filter_en.json
+1 −0 src/ov_extension.cpp
+3 −3 src/ragged_to_dense.cpp
+71 −27 src/regex_normalization.cpp
+5 −1 src/regex_normalization.hpp
+121 −42 src/sentence_piece.cpp
+17 −2 src/sentence_piece.hpp
+1 −0 src/tokenizer.hpp
+71 −0 src/utils.cpp
+10 −1 src/utils.hpp
+23 −20 src/vocab_decoder.cpp
+37 −1 tests/conftest.py
+1 −1 tests/pass_rates.json
+13,029 −0 tests/stats.json
+128 −78 tests/tokenizers_test.py
+96 −0 third-party-programs.txt

0 comments on commit 53945f7

Please sign in to comment.