Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update qwen2 tokenization #4

Merged
merged 2 commits into from
Jan 13, 2024
Merged

update qwen2 tokenization #4

merged 2 commits into from
Jan 13, 2024

Commits on Jan 13, 2024

  1. Update convert_slow_tokenizer.py

    @jklj077
    jklj077 authored Jan 13, 2024
    Configuration menu
    Copy the full SHA
    9cda260 View commit details
    Browse the repository at this point in the history

  2. add copies

    renxuancheng.rxc committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    a81deed View commit details
    Browse the repository at this point in the history