Expected closing statement token. OpenSquareBracket !== CloseStatement

[cwoolum]

System Info

"@huggingface/transformers": "^3.3.2",

Environment/Platform

• Website/web-app
• Browser extension
• Server-side (e.g., Node.js, Deno, Bun)
• Desktop app (e.g., Electron)
• Other (e.g., VSCode extension)

Description

When trying to parse messages using AutoTokenizer.from_pretrained with the model deepseek-ai/DeepSeek-R1-Distill-Llama-8B, I receive the error

Failed to get test result: Error: Parser Error: Expected closing statement token. OpenSquareBracket !== CloseStatement.

Others were seeing similar errors in other DeepSeek models but I'm not sure if that affects this package

https://huggingface.co/mlx-community/deepseek-r1-distill-qwen-1.5b/discussions/1

Reproduction

1. Load the tokenizer and try to generate a prompt

async function loadTokenizer() {
  const tokenizer = await AutoTokenizer.from_pretrained('deepseek-ai/DeepSeek-R1-Distill-Llama-8B');
  return tokenizer;
}

const tokenizer = await loadTokenizer();
const prompt = tokenizer.apply_chat_template([{ role: 'user', content: 'How are you today?' }], { tokenize: false, add_generation_prompt: true })

An error will be thrown when tokenizer.apply_chat_template is called.

[xenova]

Hi! This will be fixed by huggingface/huggingface.js#1142.

New release coming soon.