Code

update qwen2 tokenization and tests#8

Code

Merged

JustinLin610 merged 2 commits intoJustinLin610:add_qwen2JustinLin610/transformers:add_qwen2from jklj077:patch-7Copy head branch name to clipboardJan 15, 2024

+88-5

Commits

Commits on Jan 15, 2024

• update qwen2 tokenization and tests

  

  renxuancheng.rxc

  committed
  db82439

  Copy full SHA for db82439

• Update tokenization_qwen2.py

  JustinLin610

  authored
  e0c7750

  Copy full SHA for e0c7750