Code

Add the Tokenizer object logic#1874

Code

Merged

yifanmai merged 24 commits intomainstanford-crfm/helm:mainfrom joss-refactor-1-tokenizerstanford-crfm/helm:joss-refactor-1-tokenizerCopy head branch name to clipboardOct 25, 2023

+1,411-1,221

Commits

Commits on Oct 4, 2023

• Add the Tokenizer object logic

  JosselinSomervilleRoberts

  committed
  e6b9729

  Copy full SHA for e6b9729

Commits on Oct 5, 2023

• Changed most clients to use a Tokenizer object

  JosselinSomervilleRoberts

  committed
  38784f7

  Copy full SHA for 38784f7

Commits on Oct 6, 2023

• Changed remaining clients to use a Tokenizer object

  JosselinSomervilleRoberts

  committed
  daf431b

  Copy full SHA for daf431b

• Removed calls to CachableClient.tokenize()

  JosselinSomervilleRoberts

  committed
  cfd0c15

  Copy full SHA for cfd0c15

Commits on Oct 7, 2023

• Add TODOs

  JosselinSomervilleRoberts

  committed
  4bf3c46

  Copy full SHA for 4bf3c46

• Make client methods abstract

  JosselinSomervilleRoberts

  committed
  d6341ef

  Copy full SHA for d6341ef

Commits on Oct 9, 2023

• Resolve merge conflicts

  JosselinSomervilleRoberts

  committed
  d2ac135

  Copy full SHA for d2ac135

• Fix ICE Tokenizer test

  JosselinSomervilleRoberts

  committed
  4e83fd2

  Copy full SHA for 4e83fd2

• Fix Critique breaking change

  JosselinSomervilleRoberts

  committed
  0445a78

  Copy full SHA for 0445a78

• Revert fix

  JosselinSomervilleRoberts

  committed
  e1cbe32

  Copy full SHA for e1cbe32

Commits on Oct 10, 2023

• Fix all window service test issues except for Cohere

  JosselinSomervilleRoberts

  committed
  40337f7

  Copy full SHA for 40337f7

Commits on Oct 11, 2023

• Resolve merge conflicts with HuggingFace refactorization

  JosselinSomervilleRoberts

  committed
  037a869

  Copy full SHA for 037a869

• Refactor CachableClient -> CachingClient

  JosselinSomervilleRoberts

  committed
  1077699

  Copy full SHA for 1077699

• Refactor yalm_tokenizer_src -> yalm_tokenizer_data

  JosselinSomervilleRoberts

  committed
  b0fefef

  Copy full SHA for b0fefef

• Merge #1891

  JosselinSomervilleRoberts

  committed
  0e47750

  Copy full SHA for 0e47750

Commits on Oct 17, 2023

• Merge branch 'main' into joss-refactor-1-tokenizer

  JosselinSomervilleRoberts

  committed
  a907774

  Copy full SHA for a907774

• CachableTokenizer -> CachingTokenizer

  JosselinSomervilleRoberts

  committed
  c9aa4fd

  Copy full SHA for c9aa4fd

• Port VLM model Idefics to use new Tokenizer logic

  JosselinSomervilleRoberts

  committed
  b1badb7

  Copy full SHA for b1badb7

• Add TODOs to remove tokenize and decode methods from Client

  JosselinSomervilleRoberts

  committed
  57fd565

  Copy full SHA for 57fd565

• Change methods of CachingTokenizer to preserve existing Cache

  JosselinSomervilleRoberts

  committed
  50bb454

  Copy full SHA for 50bb454

Commits on Oct 18, 2023

• Change raw_request to request in _tokenization_raw_response_to_tokens and _decode_raw_response_to_text

  JosselinSomervilleRoberts

  committed
  9876162

  Copy full SHA for 9876162

• Merge branch 'main' into joss-refactor-1-tokenizer

  JosselinSomervilleRoberts

  committed
  359994c

  Copy full SHA for 359994c

• Merge branch 'main' into joss-refactor-1-tokenizer

  JosselinSomervilleRoberts

  committed
  8d2e0e2

  Copy full SHA for 8d2e0e2

• Fixing type of wrap_request

  JosselinSomervilleRoberts

  committed
  6b86de7

  Copy full SHA for 6b86de7