llama_runner crash when running HF models

[guangy10]

🐛 Describe the bug

Build and run llama_runner from latest trunk (7534d4854b94cb49ea54d0e937c5af59ab18d2a9) still crashes:
cmake-out/examples/models/llama/llama_main --model_path=smollm3/smollm3-3b-8da4w.pte --tokenizer_path=smollm3/tokenizer.json --prompt="Once upon a time"

stacktrace:

I tokenizers:regex.cpp:27] Registering override fallback regex
I tokenizers:hf_tokenizer.cpp:109] Setting up normalizer...
libc++abi: terminating due to uncaught exception of type nlohmann::json_abi_v3_11_3::detail::type_error: [json.exception.type_error.304] cannot use at() with null
Abort trap: 6

The PTE and tokenizer can be downloaded here: https://huggingface.co/pytorch/SmolLM3-3B-8da4w (need to join the organization first), the one I previously validated on-device using llama_runner built from a commit about 15 days ago.

Follow this instruction to build llama_runner: https://github.com/pytorch/executorch/blob/main/examples/models/llama/README.md#step-3-run-on-your-computer-to-validate

Versions

latest trunk (7534d4854b94cb49ea54d0e937c5af59ab18d2a9)

cc @larryliu0820 @JacobSzwejbka @lucylq

[guangy10]

cc: @jackzhxng @larryliu0820

[cavusmustafa]

I observe a similar error using Llama3.2 as well.

I tokenizers:hf_tokenizer.cpp:122] Setting up normalizer...
terminate called after throwing an instance of 'nlohmann::json_abi_v3_11_3::detail::type_error'
  what():  [json.exception.type_error.304] cannot use at() with null

The workaround for me to comment out the normalizer setup part in hf_tokenizer.cpp:
https://github.com/pytorch-labs/tokenizers/blob/23359bdce7bedc084e101c39e89506dff459dab8/src/hf_tokenizer.cpp#L108

But after that I see the error message below. The process doesn't crash with this error, it still generates the output text.

E0000 00:00:1752626580.453141 2635958 re2.cc:237] Error parsing '((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n...': invalid perl operator: (?!
E tokenizers:re2_regex.cpp:22] Failed to compile regex: ((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+), error: invalid perl operator: (?!

latest commit id: 9711976
model: https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct/tree/main

[winskuo-quic]

I was rebasing earlier today to mainline and encountered another error:
libc++abi: terminating due to uncaught exception of type std::runtime_error: Unsupported Normalizer type: NFC.

I am passing qwen2.5 tokenizer's json during runtime.
To get the tokenizer json, I did:
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
runtime_tokenizer_path = tokenizer.save_pretrained('.')[-1]

[guangy10]

I was rebasing earlier today to mainline and encountered another error: libc++abi: terminating due to uncaught exception of type std::runtime_error: Unsupported Normalizer type: NFC.

I am passing qwen2.5 tokenizer's json during runtime. To get the tokenizer json, I did: tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B") runtime_tokenizer_path = tokenizer.save_pretrained('.')[-1]

@winskuo-quic Just in case anyone is using the Android demo app (which is using the llama_runner code and tokenizier behind), I was seeing a different tokenizer error when running SmolLM3 last week:

07-11 10:02:32.193 16496 16540 E AndroidRuntime: java.lang.RuntimeException: Unsupported behavior 'Isolated' for Split PreTokenizer. Only 'MergedWithPrevious' is supported.

and after rebased I got another different:

07-11 11:46:00.832 21689 21747 E AndroidRuntime: FATAL EXCEPTION: Thread-2
07-11 11:46:00.832 21689 21747 E AndroidRuntime: Process: com.example.executorchllamademo, PID: 21689
07-11 11:46:00.832 21689 21747 E AndroidRuntime: com.facebook.jni.CppException: [json.exception.type_error.304] cannot use at() with null

[larryliu0820]

Looking

[larryliu0820]

Fix for smolLM3 meta-pytorch/tokenizers#102

[larryliu0820]

Fix for smolLM3 pytorch-labs/tokenizers#102

This fixed llama3.2 as well. Added unit test.

@cavusmustafa