SHA256 checksums correctness

[anzz1]

Not all of these checksums seem to be correct. Are they calculated with the "v2" new model format after the tokenizer change? PR: #252 Issue: #324

For example, "models/alpaca-7B/ggml-model-q4_0.bin"

v1: 1f582babc2bd56bb63b33141898748657d369fd110c4358b2bc280907882bf13
v2: 8d5562ec1d8a7cfdcf8985a9ddf353339d942c7cf52855a92c9ff59f03b541bc

The SHA256SUMS file has the old v1 hash.
Maybe using a naming scheme like "ggml2-model-q4_0.bin" would be good to differentiate between the versions and avoid confusion.

Originally posted by @anzz1 in #338 (comment)

edit: After converting the models to the new format, I found out that the "v2" hash above is also incorrect.
The sha256 for ./models/alpaca-7B-ggml/ggml-model-q4_0.bin is supposed to be 2fe0cd21df9c235c0d917c14e1b18d2d7320ed5d8abe48545518e96bb4227524

[gjmulder]

I'm still in the process of finding/converting the 7B and 13B alpaca models to ggml2

I'll then recompute all the hashes with the latest build, and also provide a file with the magic numbers and versions for each.

[Green-Sky]

the new ggml file format has the version number 1. calling it ggml2 or "v2" is going to cause confusion. the new file format switched the file magic from "ggml" to "ggmf", maybe we should lean into that.

[anzz1]

Some checksums (q4_0 and gptq-4b quantizations, new tokenizer format)

ggml-q4-checksums.zip

e: added more checksums

[gjmulder]

Some checksums (q4_0 quantization, new tokenizer format)

ggml-q4_0-checksums.zip

I'd trust your checksums for the alpaca models over mine.

$ cat SHA256SUMS.gary
alpaca-13B-ggml/ggml-model-q4_0.bin: FAILED
alpaca-13B-ggml/params.json: FAILED open or read
alpaca-13B-ggml/tokenizer.model: FAILED open or read
alpaca-30B-ggml/ggml-model-q4_0.bin: OK
alpaca-30B-ggml/params.json: OK
alpaca-30B-ggml/tokenizer.model: FAILED open or read
alpaca-7B-ggml/ggml-model-q4_0.bin: FAILED
alpaca-7B-ggml/params.json: FAILED open or read
alpaca-7B-ggml/tokenizer.model: FAILED open or read
llama-13B-ggml/ggml-model-q4_0.bin: OK
llama-13B-ggml/ggml-model-q4_0.bin.1: OK
llama-13B-ggml/params.json: OK
llama-13B-ggml/tokenizer.model: FAILED open or read
llama-30B-ggml/ggml-model-q4_0.bin: OK
llama-30B-ggml/ggml-model-q4_0.bin.1: OK
llama-30B-ggml/ggml-model-q4_0.bin.2: OK
llama-30B-ggml/ggml-model-q4_0.bin.3: OK
llama-30B-ggml/params.json: OK
llama-30B-ggml/tokenizer.model: FAILED open or read
llama-65B-ggml/ggml-model-q4_0.bin: OK
llama-65B-ggml/ggml-model-q4_0.bin.1: OK
llama-65B-ggml/ggml-model-q4_0.bin.2: OK
llama-65B-ggml/ggml-model-q4_0.bin.3: OK
llama-65B-ggml/ggml-model-q4_0.bin.4: OK
llama-65B-ggml/ggml-model-q4_0.bin.5: OK
llama-65B-ggml/ggml-model-q4_0.bin.6: OK
llama-65B-ggml/ggml-model-q4_0.bin.7: OK
llama-65B-ggml/params.json: OK
llama-65B-ggml/tokenizer.model: FAILED open or read
llama-7B-ggml/ggml-model-q4_0.bin: OK
llama-7B-ggml/params.json: OK
llama-7B-ggml/tokenizer.model: FAILED open or read

[Green-Sky]

the problem with the alpaca models is, that there are alot of different once, by different peoples.

[gjmulder]

the problem with the alpaca models is, that there are alot of different once, by different peoples.

Yes. However we're supporting them, so we need to decide what we can support.

[gjmulder]

Upvote for @anzz1's new naming convention for the various model subdirs.