Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automate vocab support and model conversion #7379

Draft
wants to merge 102 commits into
base: master
Choose a base branch
from
Draft

Automate vocab support and model conversion #7379

wants to merge 102 commits into from

Commits on May 18, 2024

  1. feat: Add prototype for managing huggingface hub content

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    dbdf6c2 View commit details
    Browse the repository at this point in the history

  2. feat: Add utils for logging and writing when interacting with Hugging… 

    …FaceHub
    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    ba13d64 View commit details
    Browse the repository at this point in the history

  3. refactor: Add log for status and fix url path variable name

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    742abeb View commit details
    Browse the repository at this point in the history

  4. patch: Apply minor fixes for handling headers and writing content

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    98cf788 View commit details
    Browse the repository at this point in the history

  5. feat: Add prototype for requesting vocab related files

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    4790f76 View commit details
    Browse the repository at this point in the history

  6. feat: Add download_model method and fix references for clarity to mit… 

    …igate confusion
    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    5c8144e View commit details
    Browse the repository at this point in the history

  7. refactor: Clean up variable names and separate concerns when download… 

    …ing tokenizers
    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    1a286c8 View commit details
    Browse the repository at this point in the history

  8. chore: Fix spacing

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    3ba01c7 View commit details
    Browse the repository at this point in the history

  9. feat: Add tokenizer types, model types, and model repos

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    f7515ab View commit details
    Browse the repository at this point in the history

  10. refactor: Apply model schema to tokenizer downloads 

    - Add imports for json and hashlib
- Add missing models: phi, stablelm, mistral, and mixtral
- Fix constructor logic
- Fix how models are accessed
- Apply model schema to download_model method
    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    3022587 View commit details
    Browse the repository at this point in the history

  11. feat: Add method for generating the checksums and writing the results… 

    … to a json file
    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    b2ca23c View commit details
    Browse the repository at this point in the history

  12. feat: Add pre-tokenizer logging

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    5eda2c9 View commit details
    Browse the repository at this point in the history

  13. refactor: Apply SoC for HF requests, vocab, and weights

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    2ef73ee View commit details
    Browse the repository at this point in the history

  14. chore: Apply isort to package gguf init

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    04fb788 View commit details
    Browse the repository at this point in the history

  15. feat: Add pre-tokenizer CLI tooling

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    832b449 View commit details
    Browse the repository at this point in the history

  16. chore: Fix line spacing

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    b6f70b8 View commit details
    Browse the repository at this point in the history

  17. chore: Fix model path references

    @teleprint-me
    teleprint-me committed May 18, 2024
    Configuration menu
    Copy the full SHA
    006bb60 View commit details
    Browse the repository at this point in the history

Commits on May 19, 2024

  1. feat: Add example script for automating generating tokenizer model ch… 

    …ecksums and tests
    @teleprint-me
    teleprint-me committed May 19, 2024
    Configuration menu
    Copy the full SHA
    1a82573 View commit details
    Browse the repository at this point in the history

  2. chore: Remove cluttered vocab files

    @teleprint-me
    teleprint-me committed May 19, 2024
    Configuration menu
    Copy the full SHA
    4b3735c View commit details
    Browse the repository at this point in the history

  3. patch: Add exception handling for non-existent vocab related files

    @teleprint-me
    teleprint-me committed May 19, 2024
    Configuration menu
    Copy the full SHA
    0479e96 View commit details
    Browse the repository at this point in the history

  4. feat: Add function for generating vocab script and fix CLI opts

    @teleprint-me
    teleprint-me committed May 19, 2024
    Configuration menu
    Copy the full SHA
    bd32266 View commit details
    Browse the repository at this point in the history

  5. feat: Add vocab generation script

    @teleprint-me
    teleprint-me committed May 19, 2024
    Configuration menu
    Copy the full SHA
    d02a0f4 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into auto-model-support

    @teleprint-me
    teleprint-me committed May 19, 2024
    Configuration menu
    Copy the full SHA
    ce777c8 View commit details
    Browse the repository at this point in the history

  7. fix: Apply fix to verbose help description and generating vocab tests… 

    … option
    @teleprint-me
    teleprint-me committed May 19, 2024
    Configuration menu
    Copy the full SHA
    da5deeb View commit details
    Browse the repository at this point in the history

  8. patch: Fix CLI option for generating vocab tests

    @teleprint-me
    teleprint-me committed May 19, 2024
    Configuration menu
    Copy the full SHA
    316b404 View commit details
    Browse the repository at this point in the history

  9. refactor: Simplify the get_vocab_base_pre method

    @teleprint-me
    teleprint-me committed May 19, 2024
    Configuration menu
    Copy the full SHA
    5840b6f View commit details
    Browse the repository at this point in the history

  10. fix: Remove dangling if statement

    @teleprint-me
    teleprint-me committed May 19, 2024
    Configuration menu
    Copy the full SHA
    dcc5d42 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2024

  1. feat: Add prototype for identifying the vocab type

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    c6f2a48 View commit details
    Browse the repository at this point in the history

  2. feat: Attempt to mirror the llama.cpp API for compatibility

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    89a46fe View commit details
    Browse the repository at this point in the history

  3. patch: Fix nested quotes for dict refs

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    a0362ea View commit details
    Browse the repository at this point in the history

  4. fix: Use __name__ as logger name

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    9a2834e View commit details
    Browse the repository at this point in the history

  5. fix: Add missing model architectures

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    381dad5 View commit details
    Browse the repository at this point in the history

  6. chore: Add english pangram to vocab tests

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    6fc4492 View commit details
    Browse the repository at this point in the history

  7. refactor: Add proper names for remote model references

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    a1951e2 View commit details
    Browse the repository at this point in the history

  8. refactor: Use proper names for referenced member variables

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    bdd0286 View commit details
    Browse the repository at this point in the history

  9. chore: Allow the user to config the logger

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    18bb36e View commit details
    Browse the repository at this point in the history

  10. refactor: Restructure tokenizer model metadata

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    d9ba963 View commit details
    Browse the repository at this point in the history

  11. chore: Move enums and model map to constants

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    2fa2c7a View commit details
    Browse the repository at this point in the history

  12. chore: Fix and update comments

    @teleprint-me
    teleprint-me committed May 20, 2024
    Configuration menu
    Copy the full SHA
    5978bb0 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. chore: Add tokenizer constants for model metadata

    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    12537fd View commit details
    Browse the repository at this point in the history

  2. proto: Add experimental vocab pre-tokenizer regular expressions

    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    aed0573 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into auto-model-support

    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    a35b767 View commit details
    Browse the repository at this point in the history

  4. chore: Apply deduped token type references

    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    6296206 View commit details
    Browse the repository at this point in the history

  5. chore: Remove unused enum import reference

    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    a3bdac0 View commit details
    Browse the repository at this point in the history

  6. feat: Add hf model mapping descriptors for each repo

    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    fb32f50 View commit details
    Browse the repository at this point in the history

  7. chore: Add formatting, set common vocab files, apply pattern to model… 

    … map
    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    4768650 View commit details
    Browse the repository at this point in the history

  8. chore: Rename from repo to model repo and reorder for improved readab… 

    …ility
    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    2fe28ad View commit details
    Browse the repository at this point in the history

  9. refactor: Rename constants to reduce confusion between references

    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    83b9fcd View commit details
    Browse the repository at this point in the history

  10. docs: Fix comment

    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    b2aac68 View commit details
    Browse the repository at this point in the history

  11. refactor: Add experimental model mappings

    @teleprint-me
    teleprint-me committed May 21, 2024
    Configuration menu
    Copy the full SHA
    34e14ae View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. refactor: Add experimental mapping for BPE pre-tokenizers

    @teleprint-me
    teleprint-me committed May 22, 2024
    Configuration menu
    Copy the full SHA
    0b43e14 View commit details
    Browse the repository at this point in the history

  2. chore: Map model file and vocab types

    @teleprint-me
    teleprint-me committed May 22, 2024
    Configuration menu
    Copy the full SHA
    12285b5 View commit details
    Browse the repository at this point in the history

  3. refactor: Simplify BPE pre-tokenizer mapping

    @teleprint-me
    teleprint-me committed May 22, 2024
    Configuration menu
    Copy the full SHA
    1957ca4 View commit details
    Browse the repository at this point in the history

  4. chore: Add model metadata

    @teleprint-me
    teleprint-me committed May 22, 2024
    Configuration menu
    Copy the full SHA
    cd00be8 View commit details
    Browse the repository at this point in the history

  5. chore: Add prototyped CLI options

    @teleprint-me
    teleprint-me committed May 22, 2024
    Configuration menu
    Copy the full SHA
    78d7828 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. feat: Add custom huggingface hub api 

    Signed-off-by: teleprint-me <77757836+teleprint-me@users.noreply.github.com>
    @teleprint-me
    teleprint-me committed May 23, 2024
    Configuration menu
    Copy the full SHA
    9814b7f View commit details
    Browse the repository at this point in the history

  2. chore: Add required vocabulary constants

    @teleprint-me
    teleprint-me committed May 23, 2024
    Configuration menu
    Copy the full SHA
    9ba6b92 View commit details
    Browse the repository at this point in the history

  3. refactor: Apply consistent naming conventions

    @teleprint-me
    teleprint-me committed May 23, 2024
    Configuration menu
    Copy the full SHA
    0ccf579 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. feat: Add CLI tool for fetching vocab files

    @teleprint-me
    teleprint-me committed May 24, 2024
    Configuration menu
    Copy the full SHA
    c92c6ad View commit details
    Browse the repository at this point in the history

  2. refactor: Simplify huggingface hub api implementation

    @teleprint-me
    teleprint-me committed May 24, 2024
    Configuration menu
    Copy the full SHA
    1749209 View commit details
    Browse the repository at this point in the history

  3. refactor: Simplify huggingface hub vocab request

    @teleprint-me
    teleprint-me committed May 24, 2024
    Configuration menu
    Copy the full SHA
    f62080a View commit details
    Browse the repository at this point in the history

  4. refactor: Apply fixes to required arguments and fixes to options

    @teleprint-me
    teleprint-me committed May 24, 2024
    Configuration menu
    Copy the full SHA
    ea4fc10 View commit details
    Browse the repository at this point in the history

  5. chore: Apply ruff formatting for readability

    @teleprint-me
    teleprint-me committed May 24, 2024
    Configuration menu
    Copy the full SHA
    b4b553f View commit details
    Browse the repository at this point in the history

  6. refactor: Add tokenizer path, add methods for extracting vocab metada… 

    …ta, fix checksum method name
    @teleprint-me
    teleprint-me committed May 24, 2024
    Configuration menu
    Copy the full SHA
    77bc739 View commit details
    Browse the repository at this point in the history

  7. refactor: Add fixes for logging

    @teleprint-me
    teleprint-me committed May 24, 2024
    Configuration menu
    Copy the full SHA
    c91dcdf View commit details
    Browse the repository at this point in the history

  8. refactor: Apply fix for file path references

    @teleprint-me
    teleprint-me committed May 24, 2024
    Configuration menu
    Copy the full SHA
    e62e09b View commit details
    Browse the repository at this point in the history

  9. refactor: Add a custom tokenizer component and fix vocab request class

    @teleprint-me
    teleprint-me committed May 24, 2024
    Configuration menu
    Copy the full SHA
    6c9ac0f View commit details
    Browse the repository at this point in the history

  10. refactor: Simplify the huggingface hub api to enable flexible model r… 

    …equests
    @teleprint-me
    teleprint-me committed May 24, 2024
    Configuration menu
    Copy the full SHA
    6409694 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2024

  1. patch: Apply fix for paths and logging

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    6da2bd6 View commit details
    Browse the repository at this point in the history

  2. refactor: Add remote repository listings to the bas HFHub class

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    168297f View commit details
    Browse the repository at this point in the history

  3. refactor: Simplify API and merge HFModel into HFHub

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    99275a1 View commit details
    Browse the repository at this point in the history

  4. refactor: Abstract file and logger management to streamline api inter… 

    …face
    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    4438d05 View commit details
    Browse the repository at this point in the history

  5. refactor: Streamline method signatures and clarify method names relat… 

    …ed to downloading repo files
    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    fda2319 View commit details
    Browse the repository at this point in the history

  6. Refactor HFubModel and HFHubTokenizer to fix reference issues

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    2ffe6b8 View commit details
    Browse the repository at this point in the history

  7. refactor: Add support for model file types

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    63c3410 View commit details
    Browse the repository at this point in the history

  8. refactor: Apply huggingface_hub api to CLI

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    6c1b011 View commit details
    Browse the repository at this point in the history

  9. docs: Add revisions to hub-vocab.py module level docstring

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    e9759de View commit details
    Browse the repository at this point in the history

  10. refactor: Add function for building and parsing CLI arguments

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    f30bd63 View commit details
    Browse the repository at this point in the history

  11. feat: Add static methods for resolving model types and model extensions

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    da72554 View commit details
    Browse the repository at this point in the history

  12. chore: Add comments for each file extension type

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    fcd20ab View commit details
    Browse the repository at this point in the history

  13. feat: Add example script for downloading models

    @teleprint-me
    teleprint-me committed May 25, 2024
    Configuration menu
    Copy the full SHA
    e4275bc View commit details
    Browse the repository at this point in the history

Commits on May 26, 2024

  1. Merge branch 'huggingface-hub-api' into auto-model-support

    @teleprint-me
    teleprint-me committed May 26, 2024
    Configuration menu
    Copy the full SHA
    b3a5429 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into auto-model-support

    @teleprint-me
    teleprint-me committed May 26, 2024
    Configuration menu
    Copy the full SHA
    36bea17 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. feat: Add experimental model registry for known models and their rela… 

    …ted metadata
    @teleprint-me
    teleprint-me committed May 27, 2024
    Configuration menu
    Copy the full SHA
    7f48eb9 View commit details
    Browse the repository at this point in the history

  2. feat: Add a proto sketch for handling mode vocab metadata

    @teleprint-me
    teleprint-me committed May 27, 2024
    Configuration menu
    Copy the full SHA
    b1c922f View commit details
    Browse the repository at this point in the history

  3. feat: Ignore pre-existing model files

    @teleprint-me
    teleprint-me committed May 27, 2024
    Configuration menu
    Copy the full SHA
    0732bd9 View commit details
    Browse the repository at this point in the history

  4. feat: Add prototype for bootstrapping registry

    @teleprint-me
    teleprint-me committed May 27, 2024
    Configuration menu
    Copy the full SHA
    2153949 View commit details
    Browse the repository at this point in the history

  5. chore: Add pre tokenizers and include enum mappings

    @teleprint-me
    teleprint-me committed May 27, 2024
    Configuration menu
    Copy the full SHA
    0a478c0 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. refactor: Simplify tokenizers implementation

    @teleprint-me
    teleprint-me committed May 28, 2024
    Configuration menu
    Copy the full SHA
    9dbc957 View commit details
    Browse the repository at this point in the history

  2. chore: Fix import path, token comparisons, and update token type refe… 

    …rences
    @teleprint-me
    teleprint-me committed May 28, 2024
    Configuration menu
    Copy the full SHA
    aa28cfe View commit details
    Browse the repository at this point in the history

  3. refactor: Simplify huggingface hub api and update to reflect changes … 

    …in constants.py
    @teleprint-me
    teleprint-me committed May 28, 2024
    Configuration menu
    Copy the full SHA
    f1d067e View commit details
    Browse the repository at this point in the history

  4. refactor: Apply updates to example script for generating the registry

    @teleprint-me
    teleprint-me committed May 28, 2024
    Configuration menu
    Copy the full SHA
    5c92809 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into auto-model-support

    @teleprint-me
    teleprint-me committed May 28, 2024
    Configuration menu
    Copy the full SHA
    6a725cf View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. Merge branch 'master' into auto-model-support

    @teleprint-me
    teleprint-me committed May 29, 2024
    Configuration menu
    Copy the full SHA
    de0f0d0 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2024

  1. Merge branch 'master' into auto-model-support

    @teleprint-me
    teleprint-me committed May 31, 2024
    Configuration menu
    Copy the full SHA
    c2e4897 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2024

  1. refactor: Add prototyped bridge interface for tokenizers and llama.cpp

    @teleprint-me
    teleprint-me committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    47ef615 View commit details
    Browse the repository at this point in the history

  2. refactor: Add prototyped bridge interface for transformers and tokeni… 

    …zers
    @teleprint-me
    teleprint-me committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    c447010 View commit details
    Browse the repository at this point in the history

  3. patch: Apply fix for backward compat for source repo

    @teleprint-me
    teleprint-me committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    647d252 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2024

  1. Merge branch 'master' into auto-model-support

    @teleprint-me
    teleprint-me committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    250bddf View commit details
    Browse the repository at this point in the history

  2. chore: Add ignore rule for generated server themes

    @teleprint-me
    teleprint-me committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    e2b7608 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'ignore-gen-themes' into auto-model-support

    @teleprint-me
    teleprint-me committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    ce8524a View commit details
    Browse the repository at this point in the history

  4. refactor: Clean up constants and simplify the custom hf hub api

    @teleprint-me
    teleprint-me committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    5836d6c View commit details
    Browse the repository at this point in the history