Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[kushalkodnad/tokenizer-registry] Introduce new registry for tokenizers #1386

Merged
merged 15 commits into from
Jul 23, 2024
Merged

[kushalkodnad/tokenizer-registry] Introduce new registry for tokenizers #1386

merged 15 commits into from
Jul 23, 2024

Commits on Jul 23, 2024

  1. Create new tokenizers registry

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    a7efff2 View commit details
    Browse the repository at this point in the history

  2. Add commented out section to build tokenizer from registry

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    3a8d570 View commit details
    Browse the repository at this point in the history

  3. Add new build_tokenizer_from_registry method for tokenizer creation

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    c8271b0 View commit details
    Browse the repository at this point in the history

  4. Register tiktoken to tokenizers registry

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    c08168e View commit details
    Browse the repository at this point in the history

  5. Update build_tokenizer() to use the tokenizers registry

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    6800bb0 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into kushalkodnad/tokenizer-registry

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    b2fb7de View commit details
    Browse the repository at this point in the history

  7. Remove unused import statement

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    e156310 View commit details
    Browse the repository at this point in the history

  8. Test out tokenizers registry functionality with dummy tokenizer

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    54f0ef9 View commit details
    Browse the repository at this point in the history

  9. Fixed pyright issue

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    ba8cac5 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'main' into kushalkodnad/tokenizer-registry

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    275804b View commit details
    Browse the repository at this point in the history

  11. Add tokenizers to expected_registry_names

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    11c5e71 View commit details
    Browse the repository at this point in the history

  12. Add tokenizers registry to __all__

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    e83772c View commit details
    Browse the repository at this point in the history

  13. Remove unnecessary parts of DummyTokenizer for test

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    c5d897f View commit details
    Browse the repository at this point in the history

  14. Merge branch 'main' into kushalkodnad/tokenizer-registry

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    afb136a View commit details
    Browse the repository at this point in the history

  15. Merge branch 'main' into kushalkodnad/tokenizer-registry

    @kushalkodn-db
    kushalkodn-db authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    3f3ce74 View commit details
    Browse the repository at this point in the history