convert : update Falcon script for new HF config

[cebtenzzre]

This aligns the Falcon convert script with the recent changes to the HuggingFace models.

[Green-Sky]

is this a continuation of #3049 ? :)

[cebtenzzre]

is this a continuation of #3049 ? :)

#3049 is a separate script that deals with 180B safetensors files. I am trying to convert 7B pytorch files using convert-falcon-hf-to-gguf.py.

By recent changes, I mean three days ago.

[Green-Sky]

Yea, but your changes are a subset of the actual changes done for 180B

$ diff convert-falcon-hf-to-gguf.py convert-falcon180-hf-to-gguf.py --color=always -w

2c2
< # HF falcon--> gguf conversion
---
> # HF falcon180B--> gguf conversion
16a17
> from safetensors import safe_open
48c49
<         if filename.startswith("pytorch_model-"):
---
>         if filename.startswith("model-00"):
90c91
< if hparams["architectures"][0] != "RWForCausalLM":
---
> if hparams["architectures"][0] != "FalconForCausalLM":
103c104
< block_count = hparams["n_layer"]
---
> block_count = hparams["num_hidden_layers"]
111,113c112,114
< gguf_writer.add_head_count(hparams["n_head"])
< if "n_head_kv" in hparams:
<     gguf_writer.add_head_count_kv(hparams["n_head_kv"])
---
> gguf_writer.add_head_count(hparams["num_attention_heads"])
> if "num_kv_heads" in hparams:
>     gguf_writer.add_head_count_kv(hparams["num_kv_heads"])
181,182c182,183
< n_head    = hparams["n_head"]
< n_head_kv = hparams["n_head_kv"] if "n_head_kv" in hparams else 1
---
> n_head    = hparams["num_attention_heads"]
> n_head_kv = hparams["num_kv_heads"] if "num_kv_heads" in hparams else 1
193c194
<         f"pytorch_model-{n:05}-of-{num_parts:05}.bin" for n in range(1, num_parts + 1)
---
>         f"model-{n:05}-of-{num_parts:05}.safetensors" for n in range(1, num_parts + 1)
200c201
<     model_part = torch.load(dir_model / part_name, map_location="cpu")
---
>     with safe_open(dir_model / part_name, framework="pt", device="cpu") as model_part:
203c204
<         data = model_part[name]
---
>             data = model_part.get_tensor(name)

(i compared an older version of convert-falcon, bc there have been some small changes on master since.)

[cebtenzzre]

Yea, but your changes are a subset of the actual changes done for 180B

Okay, so what do you suggest I do in order to get Falcon <180B working on master again?

[ggerganov]

Do we still need #3049 ?

[cebtenzzre]

Do we still need #3049 ?

The script in this PR does not know how to read safetensors files, so I think we still need that PR.

[FNsi]

By no means, why not merge #3049 first, abandon it then, finally merge this.

No one will lose the contribution records, and the falcon inference keep working.

[cebtenzzre]

Could someone test this script with Falcon-180B?

[ggerganov]

Testing atm - will take some time.
Would this change fix the reported issues in: #3484

[cebtenzzre]

Would this change fix the reported issues in: #3484

Unfortunately not. That appears to be another regression caused by #3252

[ggerganov]

Could someone test this script with Falcon-180B?

It works with this repo: https://huggingface.co/tiiuae/falcon-180B/tree/main

Which is strange, since this looks like safetensors data, which we don't expect to be able to read with this script.

[cebtenzzre]

Which is strange, since this looks like safetensors data, which we don't expect to be able to read with this script.

I recently added safetensors support to this PR so it can support 180B, which is why I asked.

[Green-Sky]

i dont have the means to test the 180b, but looks like you merged it nicely.

[rlanday]

This change breaks the old architecture (RWForCausalLM). Kind of annoying because there was another change that requires reconverting the original HuggingFace model:
#3484

[cebtenzzre]

This change breaks the old architecture (RWForCausalLM).

The official falcon-7b, falcon-40b, and falcon-180B models on HF are compatible with the new version of this script. What is your use case?

[rlanday]

I have this model downloaded from Hugging Face:
https://huggingface.co/ehartford/WizardLM-Uncensored-Falcon-40b/tree/main

Is this the official version of Falcon?
https://huggingface.co/tiiuae/falcon-7b/tree/main
The most recent commit message on the model files is “Upload RWForCausalLM.” The convert script works properly on this?

[cebtenzzre]

The most recent commit message on the model files is “Upload RWForCausalLM.” The convert script works properly on this?

The latest commit is 898df13 "Move to in-library checkpoint (for real this time)". That's the one you want.

I suppose I could add back support for the older format until the finetunes are updated.

[rlanday]

Oh, I see; they updated the config.json file but the actual multi-GB model files did not change. I think this might not be clear to some people, who might end up re-downloading the whole model again, which is not a good user experience. Also, some fine-tunes might not ever be updated. In my opinion, it would be preferable to support both formats, but I will let you decide how to prioritize this. Thank you for looking into this issue.