[Feature] Add support for StarCoder2 model architecture

[novoid]

Bug Report

Steps to Reproduce

1. Visit "Downloads"
2. Enter "starcoder2" into the search bar
3. Install a starcoder2 model such as
   • second-state/StarCoder2-15B-GGUF
   • dranger003/starcoder2-15b-GGUF
   • (Kquant03/TechxGenus-starcoder2-15b-instruct-GGUF with 29.7GB failed to download multiple times as it stuck at approximately 40 percent of the download - perhaps an issue on its own?)
4. wait for the download to finish
5. choose the new model in the UI (top center)
6. see an error message:
   • Encountered an error loading model:
   • "Could not load model due to invalid format for starcoder2-15b-Q4_0.gguf"
   • Model loading failure can happen for a variety of reasons, [...]

Besides: I can't select or copy any of those strings in the UI or error messages which makes writing those issues much more tedious as it should be.

The model download was complete and had the same md5/sha1 sum as an independent download within my shell.

Models like mistral-7b-openorca.Q4_0.gguf (downloaded weeks or even months ago) or wizardcoder-python-13b-v1.0.Q4_0.gguf (downloaded yesterday) do work.

Expected Behavior

I would expect that a completely and error-free download of a model resulted in a working chat when selected from the dropdown of models.

If the model is somehow not compatible with GPT4All, it should not be available in the list of potential downloads.

Environment

• GPT4All version: I do have the latest GPT4All v2.7.2 (as offered by the internal update mechanism)
• Operating System: Debian GNU Linux 12
• Chat model used (if applicable): see above

[manyoso]

"Besides: I can't select or copy any of those strings in the UI or error messages which makes writing those issues much more tedious as it should be."

As an aside, this is a good point. We should make it easy to copy error messages or other info messages from the UI for ease of use in bug reports and issues and the like. Thanks for the tip!

[novoid]

As an aside, this is a good point. We should make it easy to copy error messages or other info messages from the UI for ease of use in bug reports and issues and the like. Thanks for the tip!

In order not to mix up issues and capture this bug report, I created #2101 for this.

[cebtenzzre]

If the model is somehow not compatible with GPT4All, it should not be available in the list of potential downloads.

I do not know of any way to identify for certain the llama.cpp model architecture of a GGUF on HuggingFace without downloading it. I don't think GPT4All can do anything about this, people would have to standardize in some metadata on HF in the README or a JSON file in the repo. We could possibly do HTTP range requests to partially download the GGUF, but I think in practice that would be a fragile hack.

StarCoder and StarCoder2 are not models you can chat with or give instructions to - they are solely code completion models, which the chat UI is not designed for. What is your use case?

[novoid]

I had no idea.

I'm somebody with no LLM knowledge at all. I'm just curiously testing AI potential for my personal workflows. I assumed that I could use StarCoder2 to ask in the GT4all interface things like "write me a Python function that ..." or "find the bug in ..." instead of mistral which doesn't result in very reliable code so to say.

It's unfortunate that you'd have to know deeper information so that you don't have to download many GBs just to find out that this isn't working with GPT4All.

Maybe a group of people can maintain a whitelist and watch newer "models" to deal with that issue with an organizational process?

[Zibri]

same problem here I wish to use:
https://huggingface.co/TabbyML/models/resolve/main/starcoderbase-1B.Q8_0.gguf

no idea how.

[cebtenzzre]

same problem here I wish to use: huggingface.co/TabbyML/models/resolve/main/starcoderbase-1B.Q8_0.gguf

no idea how.

This is not the same issue - this is an original StarCoder model, not StarCoder 2. As described in this thread, StarCoder is not a chat model that can usefully be used with the Chat UI. But you could open a new issue to explain your situation if you'd like.