From 26da29e921dd8dd546a127c76214f20a0718f17d Mon Sep 17 00:00:00 2001 From: asafgardin <147075902+asafgardin@users.noreply.github.com> Date: Tue, 21 Nov 2023 17:37:02 +0200 Subject: [PATCH] fix: Test bump 1 (#39) * fix: crlf forbid * fix: test 1 --- .pre-commit-config.yaml | 16 +- CHANGELOG.md | 631 +++++++++++++++++---------------- examples/jurassic_tokenizer.py | 2 +- 3 files changed, 324 insertions(+), 325 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index ee298c5..0e5124b 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -16,6 +16,12 @@ repos: - id: check-symlinks - id: detect-private-key - id: no-commit-to-branch + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v4.4.0 + hooks: + - id: trailing-whitespace + - id: end-of-file-fixer + - id: mixed-line-ending - repo: https://github.com/jumanjihouse/pre-commit-hooks rev: 3.0.0 hooks: @@ -118,13 +124,3 @@ repos: entry: hadolint/hadolint:v2.10.0 hadolint types: - dockerfile - - repo: https://github.com/pre-commit/pre-commit-hooks - rev: v4.4.0 - hooks: - - id: trailing-whitespace - - id: end-of-file-fixer - - id: mixed-line-ending - - repo: https://github.com/Lucas-C/pre-commit-hooks - rev: v1.5.4 - hooks: - - id: remove-crlf diff --git a/CHANGELOG.md b/CHANGELOG.md index b7fc45c..b252550 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,413 +1,416 @@ # CHANGELOG - - ## v0.3.0 (2023-11-21) ### Documentation -* docs: Release md update before publish (#36) +- docs: Release md update before publish (#36) + +- fix: Added support for both str and path + +- fix: rename package + +- fix: updated pre commits and added new one + +- docs: Updated docs + +- ci: down grade + +- docs: Added another example ([`18ccbeb`](https://github.com/AI21Labs/ai21-tokenizer/commit/18ccbeb89745491fa2d5ac92b8e017ee2af4ca88)) + +- docs: CONTRIBUTING.md (#35) -* fix: Added support for both str and path - -* fix: rename package - -* fix: updated pre commits and added new one - -* docs: Updated docs - -* ci: down grade - -* docs: Added another example ([`18ccbeb`](https://github.com/AI21Labs/ai21-tokenizer/commit/18ccbeb89745491fa2d5ac92b8e017ee2af4ca88)) +- docs: CONTRIBUTING.md -* docs: CONTRIBUTING.md (#35) +- ci: end_of_line fix -* docs: CONTRIBUTING.md - -* ci: end_of_line fix - -* docs: inv test ([`e282440`](https://github.com/AI21Labs/ai21-tokenizer/commit/e2824402e375aa4e4714fb1afa3c212abd276c2f)) +- docs: inv test ([`e282440`](https://github.com/AI21Labs/ai21-tokenizer/commit/e2824402e375aa4e4714fb1afa3c212abd276c2f)) ### Feature -* feat: Added char for testing (#37) ([`40d3feb`](https://github.com/AI21Labs/ai21-tokenizer/commit/40d3febf9a4df9f54b91e50981326b71ce362c89)) +- feat: Added char for testing (#37) ([`40d3feb`](https://github.com/AI21Labs/ai21-tokenizer/commit/40d3febf9a4df9f54b91e50981326b71ce362c89)) ### Fix -* fix: string example (#38) ([`833038c`](https://github.com/AI21Labs/ai21-tokenizer/commit/833038c0ff348cfa240764346e2faffea09ed6ac)) - +- fix: string example (#38) ([`833038c`](https://github.com/AI21Labs/ai21-tokenizer/commit/833038c0ff348cfa240764346e2faffea09ed6ac)) ## v0.2.0 (2023-11-21) ### Chore -* chore(release): v0.2.0 [skip ci] ([`8988faa`](https://github.com/AI21Labs/ai21-tokenizer/commit/8988faa1419069916aedcb317e7c813a57a7d03d)) - -* chore(deps-dev): bump black from 22.12.0 to 23.3.0 (#32) - -Bumps [black](https://github.com/psf/black) from 22.12.0 to 23.3.0. -- [Release notes](https://github.com/psf/black/releases) -- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) -- [Commits](https://github.com/psf/black/compare/22.12.0...23.3.0) - ---- -updated-dependencies: -- dependency-name: black - dependency-type: direct:development - update-type: version-update:semver-major -... - -Signed-off-by: dependabot[bot] <support@github.com> -Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> +- chore(release): v0.2.0 [skip ci] ([`8988faa`](https://github.com/AI21Labs/ai21-tokenizer/commit/8988faa1419069916aedcb317e7c813a57a7d03d)) + +- chore(deps-dev): bump black from 22.12.0 to 23.3.0 (#32) + +Bumps [black](https://github.com/psf/black) from 22.12.0 to 23.3.0. + +- [Release notes](https://github.com/psf/black/releases) +- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) +- [Commits](https://github.com/psf/black/compare/22.12.0...23.3.0) + +--- + +updated-dependencies: + +- dependency-name: black + dependency-type: direct:development + update-type: version-update:semver-major + ... + +Signed-off-by: dependabot[bot] <support@github.com> +Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: asafgardin <147075902+asafgardin@users.noreply.github.com> ([`bb4986e`](https://github.com/AI21Labs/ai21-tokenizer/commit/bb4986e880f24eebc7be7eee050d85c4d56a1aa9)) ### Feature -* feat: Tokenizer factory (#31) - -* feat: Added tokenizer abc and factory - -* fix: api to receive default and none - -* fix: example - -* fix: factory and tests - -* fix: rename base - -* fix: rename base class - -* fix: rename package - -* fix: example - -* fix: readme and tasks - -* docs: factory class - -* docs: renames - -* fix: directory hierarchy in tests - -* fix: rename package - -* chore(release): v0.1.2 [skip ci] - -* fix: rename package - -* ci: example - -* fix: assert in example - -* fix: src_path - ---------- - +- feat: Tokenizer factory (#31) + +- feat: Added tokenizer abc and factory + +- fix: api to receive default and none + +- fix: example + +- fix: factory and tests + +- fix: rename base + +- fix: rename base class + +- fix: rename package + +- fix: example + +- fix: readme and tasks + +- docs: factory class + +- docs: renames + +- fix: directory hierarchy in tests + +- fix: rename package + +- chore(release): v0.1.2 [skip ci] + +- fix: rename package + +- ci: example + +- fix: assert in example + +- fix: src_path + +--- + Co-authored-by: github-actions <github-actions@github.com> ([`e55cd1d`](https://github.com/AI21Labs/ai21-tokenizer/commit/e55cd1dac5ad501a0a48c369a28d061f37950f5f)) ### Fix -* fix: token name (#34) ([`2b229b2`](https://github.com/AI21Labs/ai21-tokenizer/commit/2b229b28ace8ac72dcc9bc727d187350020a2e12)) - +- fix: token name (#34) ([`2b229b2`](https://github.com/AI21Labs/ai21-tokenizer/commit/2b229b28ace8ac72dcc9bc727d187350020a2e12)) ## v0.1.2 (2023-11-21) ### Chore -* chore(release): v0.1.2 [skip ci] ([`5b1dc14`](https://github.com/AI21Labs/ai21-tokenizer/commit/5b1dc140213615bbe8a6a65caea8eea2ec7d3cbc)) - -* chore(deps-dev): bump safety from 2.3.4 to 2.3.5 (#28) - -Bumps [safety](https://github.com/pyupio/safety) from 2.3.4 to 2.3.5. -- [Release notes](https://github.com/pyupio/safety/releases) -- [Changelog](https://github.com/pyupio/safety/blob/main/CHANGELOG.md) -- [Commits](https://github.com/pyupio/safety/compare/2.3.4...2.3.5) - ---- -updated-dependencies: -- dependency-name: safety - dependency-type: direct:development - update-type: version-update:semver-patch -... - -Signed-off-by: dependabot[bot] <support@github.com> +- chore(release): v0.1.2 [skip ci] ([`5b1dc14`](https://github.com/AI21Labs/ai21-tokenizer/commit/5b1dc140213615bbe8a6a65caea8eea2ec7d3cbc)) + +- chore(deps-dev): bump safety from 2.3.4 to 2.3.5 (#28) + +Bumps [safety](https://github.com/pyupio/safety) from 2.3.4 to 2.3.5. + +- [Release notes](https://github.com/pyupio/safety/releases) +- [Changelog](https://github.com/pyupio/safety/blob/main/CHANGELOG.md) +- [Commits](https://github.com/pyupio/safety/compare/2.3.4...2.3.5) + +--- + +updated-dependencies: + +- dependency-name: safety + dependency-type: direct:development + update-type: version-update:semver-patch + ... + +Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> ([`28118ac`](https://github.com/AI21Labs/ai21-tokenizer/commit/28118acd7477a128eecf36119d60d50fa1fbb169)) -* chore(deps-dev): bump pytest-mock from 3.10.0 to 3.11.1 (#24) - -Bumps [pytest-mock](https://github.com/pytest-dev/pytest-mock) from 3.10.0 to 3.11.1. -- [Release notes](https://github.com/pytest-dev/pytest-mock/releases) -- [Changelog](https://github.com/pytest-dev/pytest-mock/blob/main/CHANGELOG.rst) -- [Commits](https://github.com/pytest-dev/pytest-mock/compare/v3.10.0...v3.11.1) - ---- -updated-dependencies: -- dependency-name: pytest-mock - dependency-type: direct:development - update-type: version-update:semver-minor -... - -Signed-off-by: dependabot[bot] <support@github.com> -Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> +- chore(deps-dev): bump pytest-mock from 3.10.0 to 3.11.1 (#24) + +Bumps [pytest-mock](https://github.com/pytest-dev/pytest-mock) from 3.10.0 to 3.11.1. + +- [Release notes](https://github.com/pytest-dev/pytest-mock/releases) +- [Changelog](https://github.com/pytest-dev/pytest-mock/blob/main/CHANGELOG.rst) +- [Commits](https://github.com/pytest-dev/pytest-mock/compare/v3.10.0...v3.11.1) + +--- + +updated-dependencies: + +- dependency-name: pytest-mock + dependency-type: direct:development + update-type: version-update:semver-minor + ... + +Signed-off-by: dependabot[bot] <support@github.com> +Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: asafgardin <147075902+asafgardin@users.noreply.github.com> ([`0a36f21`](https://github.com/AI21Labs/ai21-tokenizer/commit/0a36f21e0c375b12f743e4c89c90b5391502d2b3)) ### Ci -* ci: Remove install from publish (#29) +- ci: Remove install from publish (#29) -* ci: Removed install dependency - -* ci: changlog changes ([`f8e8392`](https://github.com/AI21Labs/ai21-tokenizer/commit/f8e8392c717d93d1a103d1eec84b7580beaa1b14)) +- ci: Removed install dependency -* ci: dependabot pr limit (#27) +- ci: changlog changes ([`f8e8392`](https://github.com/AI21Labs/ai21-tokenizer/commit/f8e8392c717d93d1a103d1eec84b7580beaa1b14)) -* ci: dependabot pr limit - -* ci: dependabot pr limit ([`9b2c4f8`](https://github.com/AI21Labs/ai21-tokenizer/commit/9b2c4f885a5f2be3076f3c4563de33efb3f7af9a)) +- ci: dependabot pr limit (#27) -### Fix +- ci: dependabot pr limit + +- ci: dependabot pr limit ([`9b2c4f8`](https://github.com/AI21Labs/ai21-tokenizer/commit/9b2c4f885a5f2be3076f3c4563de33efb3f7af9a)) -* fix: workflow dispatch for release action (#33) ([`f81b4ab`](https://github.com/AI21Labs/ai21-tokenizer/commit/f81b4abbe4fe45163dec45dfce89677fe673e46b)) +### Fix +- fix: workflow dispatch for release action (#33) ([`f81b4ab`](https://github.com/AI21Labs/ai21-tokenizer/commit/f81b4abbe4fe45163dec45dfce89677fe673e46b)) ## v0.1.1 (2023-11-20) ### Chore -* chore(release): v0.1.1 [skip ci] ([`f5150f5`](https://github.com/AI21Labs/ai21-tokenizer/commit/f5150f5473bb95759cf389db2033789f69e3ac38)) +- chore(release): v0.1.1 [skip ci] ([`f5150f5`](https://github.com/AI21Labs/ai21-tokenizer/commit/f5150f5473bb95759cf389db2033789f69e3ac38)) ### Fix -* fix: used PAT (#26) - -* test: write to main - -* fix: token - -* test: debug - -* test: debugging tokens - -* test: uncomment - -* test: write to main - -* fix: token - -* test: debug - -* test: debugging tokens - -* test: uncomment - -* fix: Changed to main ([`2ff12c9`](https://github.com/AI21Labs/ai21-tokenizer/commit/2ff12c9b4d490eec0d089b18a6bdabda7e23e952)) +- fix: used PAT (#26) + +- test: write to main + +- fix: token + +- test: debug + +- test: debugging tokens + +- test: uncomment +- test: write to main + +- fix: token + +- test: debug + +- test: debugging tokens + +- test: uncomment + +- fix: Changed to main ([`2ff12c9`](https://github.com/AI21Labs/ai21-tokenizer/commit/2ff12c9b4d490eec0d089b18a6bdabda7e23e952)) ## v0.1.0 (2023-11-20) ### Chore -* chore(release): v0.1.0 [skip ci] ([`48607f9`](https://github.com/AI21Labs/ai21-tokenizer/commit/48607f92d977ce4d725879d5ac1306c8efd3255c)) +- chore(release): v0.1.0 [skip ci] ([`48607f9`](https://github.com/AI21Labs/ai21-tokenizer/commit/48607f92d977ce4d725879d5ac1306c8efd3255c)) ### Ci -* ci: Create dependabot.yml (#19) +- ci: Create dependabot.yml (#19) + +- ci: Create dependabot.yml -* ci: Create dependabot.yml - -* fix: commit-message prefix - -* fix: Added more config to dependabot action ([`23faaa8`](https://github.com/AI21Labs/ai21-tokenizer/commit/23faaa81d37a673d56542b367470aab6013d2b39)) +- fix: commit-message prefix + +- fix: Added more config to dependabot action ([`23faaa8`](https://github.com/AI21Labs/ai21-tokenizer/commit/23faaa81d37a673d56542b367470aab6013d2b39)) ### Feature -* feat: Pypi publish (#18) +- feat: Pypi publish (#18) + +- feast: Added setup.py -* feast: Added setup.py - -* feast: Added publish.yaml ([`77ee751`](https://github.com/AI21Labs/ai21-tokenizer/commit/77ee751b6d7610c4b4955ad488579a88f47d4f8b)) +- feast: Added publish.yaml ([`77ee751`](https://github.com/AI21Labs/ai21-tokenizer/commit/77ee751b6d7610c4b4955ad488579a88f47d4f8b)) -* feat: test PAT (#9) ([`a10b6a4`](https://github.com/AI21Labs/ai21-tokenizer/commit/a10b6a494c20e8929dc3ba355527f8290f27afbf)) +- feat: test PAT (#9) ([`a10b6a4`](https://github.com/AI21Labs/ai21-tokenizer/commit/a10b6a494c20e8929dc3ba355527f8290f27afbf)) -* feat: Added semantic prs actions (#8) ([`afab5ff`](https://github.com/AI21Labs/ai21-tokenizer/commit/afab5ff05154da698120d67f7a018907c6ed1ecc)) +- feat: Added semantic prs actions (#8) ([`afab5ff`](https://github.com/AI21Labs/ai21-tokenizer/commit/afab5ff05154da698120d67f7a018907c6ed1ecc)) ### Fix -* fix: Added permissions (#25) +- fix: Added permissions (#25) -* fix: Added permissions - -* fix: permissions location - -* fix: verbose - -* fix: Removed bad input "root_options" ([`d684575`](https://github.com/AI21Labs/ai21-tokenizer/commit/d684575675e85e70af96639cc3c33f125770e7da)) +- fix: Added permissions -* fix: Change token (#17) +- fix: permissions location -* fix: token key - -* fix: token github - -* fix: token github cls - -* fix: token github cls ([`ec4f35b`](https://github.com/AI21Labs/ai21-tokenizer/commit/ec4f35bf477a602e648f1d6d212c2a388b9c97e0)) +- fix: verbose -* fix: Change token (#16) +- fix: Removed bad input "root_options" ([`d684575`](https://github.com/AI21Labs/ai21-tokenizer/commit/d684575675e85e70af96639cc3c33f125770e7da)) -* fix: token key - -* fix: token github ([`d876a43`](https://github.com/AI21Labs/ai21-tokenizer/commit/d876a43b4bde91f202f93fb22411df088af49293)) +- fix: Change token (#17) -* fix: token key (#15) ([`0b76344`](https://github.com/AI21Labs/ai21-tokenizer/commit/0b76344bfcad9c03cc50d19ad2ecdf5d2127b3e2)) +- fix: token key -* fix: keys (#14) ([`b064ea6`](https://github.com/AI21Labs/ai21-tokenizer/commit/b064ea657945e7e3201d9b7678e2551ad0c75f89)) +- fix: token github -* fix: Test token (#11) +- fix: token github cls -* feat: test PAT - -* feat: test github token - -* fix: PAT ([`94f64b6`](https://github.com/AI21Labs/ai21-tokenizer/commit/94f64b6bded3bba3e6f616cfd37315bacf68b3f4)) +- fix: token github cls ([`ec4f35b`](https://github.com/AI21Labs/ai21-tokenizer/commit/ec4f35bf477a602e648f1d6d212c2a388b9c97e0)) -* fix: Test token (#10) +- fix: Change token (#16) -* feat: test PAT - -* feat: test github token ([`52484fe`](https://github.com/AI21Labs/ai21-tokenizer/commit/52484fe2a0951652ed4d34c17cf48ca47669ab11)) +- fix: token key -* fix: root_options verbose (#6) ([`220ba5b`](https://github.com/AI21Labs/ai21-tokenizer/commit/220ba5b47e9f2c4b3db0c38291abe4a983b35c2e)) +- fix: token github ([`d876a43`](https://github.com/AI21Labs/ai21-tokenizer/commit/d876a43b4bde91f202f93fb22411df088af49293)) -* fix: Release action test (#5) +- fix: token key (#15) ([`0b76344`](https://github.com/AI21Labs/ai21-tokenizer/commit/0b76344bfcad9c03cc50d19ad2ecdf5d2127b3e2)) + +- fix: keys (#14) ([`b064ea6`](https://github.com/AI21Labs/ai21-tokenizer/commit/b064ea657945e7e3201d9b7678e2551ad0c75f89)) + +- fix: Test token (#11) + +- feat: test PAT + +- feat: test github token + +- fix: PAT ([`94f64b6`](https://github.com/AI21Labs/ai21-tokenizer/commit/94f64b6bded3bba3e6f616cfd37315bacf68b3f4)) + +- fix: Test token (#10) + +- feat: test PAT + +- feat: test github token ([`52484fe`](https://github.com/AI21Labs/ai21-tokenizer/commit/52484fe2a0951652ed4d34c17cf48ca47669ab11)) + +- fix: root_options verbose (#6) ([`220ba5b`](https://github.com/AI21Labs/ai21-tokenizer/commit/220ba5b47e9f2c4b3db0c38291abe4a983b35c2e)) + +- fix: Release action test (#5) + +- fix: Added release step + +- fix: branch name for testing + +- fix: Removed comment + +- chore(release): v0.0.1 [skip ci] + +- fix: branch rename + +- fix: Removed CHANGELOG.md + +--- -* fix: Added release step - -* fix: branch name for testing - -* fix: Removed comment - -* chore(release): v0.0.1 [skip ci] - -* fix: branch rename - -* fix: Removed CHANGELOG.md - ---------- - Co-authored-by: github-actions <github-actions@github.com> ([`fae5423`](https://github.com/AI21Labs/ai21-tokenizer/commit/fae5423193754d4e60826e1a74dfb70e6e463c47)) ### Test -* test: Test token (#13) - -* feat: test PAT - -* feat: test github token - -* fix: PAT - -* test: Added test step - -* test: Added test step - -* test: Added token to use ([`d445e36`](https://github.com/AI21Labs/ai21-tokenizer/commit/d445e3688412ca4142c9309b28de6400a1f54220)) - -* test: Test token (#12) - -* feat: test PAT - -* feat: test github token - -* fix: PAT - -* test: Added test step - -* test: Added test step ([`9619c3e`](https://github.com/AI21Labs/ai21-tokenizer/commit/9619c3e064e0014d2dbf776a7bb18cd7a914c817)) +- test: Test token (#13) + +- feat: test PAT + +- feat: test github token + +- fix: PAT + +- test: Added test step + +- test: Added test step + +- test: Added token to use ([`d445e36`](https://github.com/AI21Labs/ai21-tokenizer/commit/d445e3688412ca4142c9309b28de6400a1f54220)) + +- test: Test token (#12) + +- feat: test PAT + +- feat: test github token + +- fix: PAT + +- test: Added test step + +- test: Added test step ([`9619c3e`](https://github.com/AI21Labs/ai21-tokenizer/commit/9619c3e064e0014d2dbf776a7bb18cd7a914c817)) ### Unknown -* Add kwargs to functions (#7) - -* feat: added kwargs - -* test: Added tests ([`efecff9`](https://github.com/AI21Labs/ai21-tokenizer/commit/efecff97f0291a0f155208ba09c83dfdcfd9247c)) - -* Release action (#4) - -* feat: Added release action - -* fix: Removed unnecessary code - -* fix: testing on branch - -* fix: removed node install - -* fix: Removed unnecessary step - -* fix: base-branch - -* fix: removed code - -* 0.0.2 - -* feat: python-semantic-release test - -* fix: branch name - -* fix: branch name in .toml - -* fix: change from branch to match - -* fix: Added release_action part - -* chore(release): v0.1.0 [skip ci] - -* refactor: removed CHANGELOG.md - -* fix: branch to main - -* feat: Added version.py to version_variable - -* feat: Upgraded python-semantic-release - -* feat: Added python-semantic-release - -* fix: Removed unnecessary file - -* fix: Changed version - ---------- - -Co-authored-by: github-action <41898282+github-actions[bot]@users.noreply.github.com> +- Add kwargs to functions (#7) + +- feat: added kwargs + +- test: Added tests ([`efecff9`](https://github.com/AI21Labs/ai21-tokenizer/commit/efecff97f0291a0f155208ba09c83dfdcfd9247c)) + +- Release action (#4) + +- feat: Added release action + +- fix: Removed unnecessary code + +- fix: testing on branch + +- fix: removed node install + +- fix: Removed unnecessary step + +- fix: base-branch + +- fix: removed code + +- 0.0.2 + +- feat: python-semantic-release test + +- fix: branch name + +- fix: branch name in .toml + +- fix: change from branch to match + +- fix: Added release_action part + +- chore(release): v0.1.0 [skip ci] + +- refactor: removed CHANGELOG.md + +- fix: branch to main + +- feat: Added version.py to version_variable + +- feat: Upgraded python-semantic-release + +- feat: Added python-semantic-release + +- fix: Removed unnecessary file + +- fix: Changed version + +--- + +Co-authored-by: github-action <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> ([`694683a`](https://github.com/AI21Labs/ai21-tokenizer/commit/694683ac39330944368a1e6ca3370857804bb960)) -* Add code (#2) - -* feat: Jurassic tokenizer - -* fix: remove is_start - -* fix: add types - -* fix: add types - -* chore: extracted utils - -* fix: simplified tokenizer even more - -* fix: simplified tokenizer even more - -* feat: Added tests - -* feat: exposed prop - ---------- - +- Add code (#2) + +- feat: Jurassic tokenizer + +- fix: remove is_start + +- fix: add types + +- fix: add types + +- chore: extracted utils + +- fix: simplified tokenizer even more + +- fix: simplified tokenizer even more + +- feat: Added tests + +- feat: exposed prop + +--- + Co-authored-by: Asaf Gardin <asafg@ai21.com> ([`6b80a05`](https://github.com/AI21Labs/ai21-tokenizer/commit/6b80a05549267e44e59f8ae40a92e4de68df979c)) -* First commit (#1) +- First commit (#1) -* feat: init project ([`f50565e`](https://github.com/AI21Labs/ai21-tokenizer/commit/f50565eeb8d7259dec565f1292592d66b479f962)) +- feat: init project ([`f50565e`](https://github.com/AI21Labs/ai21-tokenizer/commit/f50565eeb8d7259dec565f1292592d66b479f962)) -* Initial commit ([`556d3e6`](https://github.com/AI21Labs/ai21-tokenizer/commit/556d3e64af018cf7269d9face370099900aec8eb)) +- Initial commit ([`556d3e6`](https://github.com/AI21Labs/ai21-tokenizer/commit/556d3e64af018cf7269d9face370099900aec8eb)) diff --git a/examples/jurassic_tokenizer.py b/examples/jurassic_tokenizer.py index 186f59c..0f1d121 100644 --- a/examples/jurassic_tokenizer.py +++ b/examples/jurassic_tokenizer.py @@ -9,7 +9,7 @@ config = load_json(config_path) tokenizer = JurassicTokenizer(model_path=model_path, config=config) -example_sentence = "This sentence should be encoded and then decoded. Hurray!!!" +example_sentence = "This sentence should be encoded and then decoded. Hurray!!!!" encoded = tokenizer.encode(example_sentence) decoded = tokenizer.decode(encoded)