std::out_of_range: unordered_map::at: key not found while running deepseek-coder-33b-instruct

[Kangmo]

Please include information about your system, the steps to reproduce the bug, and the version of llama.cpp that you are using. If possible, please provide a minimal code example that reproduces the bug.

Macbook M1 pro max (64G Mem)
built llama with Release mode, using cmake.

If the bug concerns the server, please try to reproduce it first using the server test scenario framework.

reproduction steps:

git clone https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct
python ./convert.py --pad-vocab deepseek-coder-33b-instruct
./build/bin/quantize ./deepseek-coder-33b-instruct/ggml-model-f32.gguf ./deepseek-coder-33b-instruct/ggml-model-Q8_0 gguf Q8_0

execution log

Log start
main: build = 2483 (1943c019)
main: built with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.3.0
main: seed  = 10281
llama_model_loader: loaded meta data with 26 key-value pairs and 561 tensors from ./deepseek-coder-33b-instruct/ggml-model-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = .
llama_model_loader: - kv   2:                           llama.vocab_size u32              = 32256
llama_model_loader: - kv   3:                       llama.context_length u32              = 16384
llama_model_loader: - kv   4:                     llama.embedding_length u32              = 7168
llama_model_loader: - kv   5:                          llama.block_count u32              = 62
llama_model_loader: - kv   6:                  llama.feed_forward_length u32              = 19200
llama_model_loader: - kv   7:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   8:                 llama.attention.head_count u32              = 56
llama_model_loader: - kv   9:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  10:     llama.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  11:                       llama.rope.freq_base f32              = 100000.000000
llama_model_loader: - kv  12:                    llama.rope.scaling.type str              = linear
llama_model_loader: - kv  13:                  llama.rope.scaling.factor f32              = 4.000000
llama_model_loader: - kv  14:                          general.file_type u32              = 7
llama_model_loader: - kv  15:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,32256]   = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  17:                      tokenizer.ggml.scores arr[f32,32256]   = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  18:                  tokenizer.ggml.token_type arr[i32,32256]   = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 32013
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 32021
llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 32014
llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  125 tensors
llama_model_loader: - type q8_0:  436 tensors
llm_load_vocab: SPM vocabulary, but newline token not found: unordered_map::at: key not found! Using special_pad_id instead.llm_load_vocab: mismatch in special tokens definition ( 243/32256 vs 256/32256 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32256
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 16384
llm_load_print_meta: n_embd           = 7168
llm_load_print_meta: n_head           = 56
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 62
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 7
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 19200
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 100000.0
llm_load_print_meta: freq_scale_train = 0.25
llm_load_print_meta: n_yarn_orig_ctx  = 16384
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q8_0
llm_load_print_meta: model params     = 33.34 B
llm_load_print_meta: model size       = 33.00 GiB (8.50 BPW) 
llm_load_print_meta: general.name     = .
llm_load_print_meta: BOS token        = 32013 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token        = 32021 '<|EOT|>'
llm_load_print_meta: UNK token        = 0 '!'
llm_load_print_meta: PAD token        = 32014 '<｜end▁of▁sentence｜>'
llm_load_tensors: ggml ctx size =    0.43 MiB
ggml_backend_metal_buffer_from_ptr: allocated buffer, size = 33554.00 MiB, (33554.06 / 49152.00)
llm_load_tensors: offloading 62 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 63/63 layers to GPU
llm_load_tensors:        CPU buffer size =   234.28 MiB
llm_load_tensors:      Metal buffer size = 33553.99 MiB
....................................................................................................
llama_new_context_with_model: n_ctx      = 4096
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: freq_base  = 100000.0
llama_new_context_with_model: freq_scale = 0.25
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Max
ggml_metal_init: picking default device: Apple M1 Max
ggml_metal_init: loading '/Users/kangmo/src/llama.cpp/build/bin/default.metallib'
ggml_metal_init: GPU name:   Apple M1 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple7  (1007)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 51539.61 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   992.00 MiB, (34547.88 / 49152.00)
llama_kv_cache_init:      Metal KV buffer size =   992.00 MiB
llama_new_context_with_model: KV self size  =  992.00 MiB, K (f16):  496.00 MiB, V (f16):  496.00 MiB
llama_new_context_with_model:        CPU  output buffer size =   252.00 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   512.02 MiB, (35059.89 / 49152.00)
llama_new_context_with_model:      Metal compute buffer size =   512.00 MiB
llama_new_context_with_model:        CPU compute buffer size =    22.00 MiB
llama_new_context_with_model: graph nodes  = 2050
llama_new_context_with_model: graph splits = 2

system_info: n_threads = 10 / 10 | AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | 
libc++abi: terminating due to uncaught exception of type std::out_of_range: unordered_map::at: key not found
zsh: abort      ./build/bin/main -m ./deepseek-coder-33b-instruct/ggml-model-Q8_0.gguf --seed

[dranger003]

I believe this is related: #5981

[0x4E69676874466F78]

workaround #5973

[Kangmo]

workaround #5973

Thank you so much