Code

Add an abstract class for Tokenizer#53

Code

Merged

bhavya01 merged 13 commits intoAI-Hypercomputer:mainAI-Hypercomputer/JetStream:mainfrom bhavya01:tokenizerbhavya01/JetStream:tokenizerCopy head branch name to clipboardApr 26, 2024

+180-12

Commits

Commits on Apr 24, 2024

• Add an abstract class for tokenizer

  bhavya01

  committed
  5ef54d4

  Copy full SHA for 5ef54d4

• Add sentence piece tokenizer as a subclass of Tokenizer

  bhavya01

  committed
  d829d61

  Copy full SHA for d829d61

• Fix decode method for SentencePieceTokenizer

  bhavya01

  committed
  4ff58b7

  Copy full SHA for 4ff58b7

• Merge branch 'google:main' into tokenizer

  bhavya01

  authored
  a506f29

  Copy full SHA for a506f29

• Fix circular import issue

  bhavya01

  committed
  54d98e4

  Copy full SHA for 54d98e4

• fix type annotations

  bhavya01

  committed
  f669516

  Copy full SHA for f669516

• fix linting issues

  bhavya01

  committed
  1ef30e8

  Copy full SHA for 1ef30e8

• Format files using pyink

  bhavya01

  committed
  807e4e9

  Copy full SHA for 807e4e9

Commits on Apr 25, 2024

• Merge branch 'google:main' into tokenizer

  bhavya01

  authored
  67f9c00

  Copy full SHA for 67f9c00

• Update the tokenizer decode interface to return ids instead of str

  bhavya01

  committed
  f8e1e4e

  Copy full SHA for f8e1e4e

• format using pyink

  bhavya01

  committed
  97ab430

  Copy full SHA for 97ab430

Commits on Apr 26, 2024

• Move Tokenizer class to a tokenizer_api.py file

  bhavya01

  committed
  3c14c3d

  Copy full SHA for 3c14c3d

• Update engine.build_tokenizer method to return SentencePieceTokenizer by default

  bhavya01

  committed
  3d65556

  Copy full SHA for 3d65556