Discussions

• 2 You must be logged in to vote

  💬

  llama.cpp tokenization is not the same as Meta tokenization.

  viniciusarruda started Jul 21, 2023 in General

  

  

  1
• 1 You must be logged in to vote

  💬

  llama.cpp is missing support for special tokens for tokenizing bos/eos

  ycros started Jul 21, 2023 in General

  

  

  1