"ï¸ı" is causing chktok to mismatch when using chkhsh

[arch-btw]

I think the comma is not being escaped.

convert-hf-to-gguf.py : loses the token
convert-hf-to-gguf-update.py: adds token correctly

Related: #7018

Tested with Falcon and Qwen2, both fail on the same token.

[arch-btw]

Same problem: #7022

@ggerganov

[arch-btw]

convert-hf-to-gguf-update.py:

convert-hf-to-gguf.py

[teleprint-me]

@arch-btw What are the sources? Need the links to investigate.

[arch-btw]

Thank you @teleprint-me the sources are:

https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat

and

https://huggingface.co/tiiuae/falcon-7b

[teleprint-me]

I have to go to work—have a double today—but I'll check it out between shifts.

[ggerganov]

I don't understand - which comma is this related to?
Are you sure you are using the two scripts with the same tokenizer?

[teleprint-me]

@ggerganov The arrays in the output are not equal. There are missing vectors encodings. The missing encodings change the hash output. I don't have time to do it between shifts, but I can take a closer look tonight. Juggling too many things rn.

[arch-btw]

@ggerganov Thanks, yes, I'm using the same tokenizers. It's possible that it's not the comma (see below) but something is causing convert-hf-to-gguf.py to not include this token. I think there's something strange about these symbol(s): ï¸ı that's causing convert-hf-to-gguf.py to not include it, I thought maybe the comma is being interpreted literally.

I meant this symbol here, it's some unusual type of comma:

[arch-btw]

Ok I think I'm getting a bit closer, it's actually the character after the comma.

https://en.wikipedia.org/wiki/Dotted_and_dotless_I_in_computing?lang=en

Update:

the ı symbol is not part of latin-1 as used here: https://github.com/ggerganov/llama.cpp/blob/master/convert-hf-to-gguf.py#L1805C71-L1805C78

[teleprint-me]

@arch-btw What OS are you using when you attempt to convert?

[arch-btw]

@teleprint-me , attempting to convert using arch linux. Do you think it might be OS related?