Using GPTeam with gpt4all or llama-cpp-python openai-compatible API endpoint

[dflatline]

Hi!

I got GPTeam to connect to the GPT4all local chat server mode by hacking the openai.api_base url to point at the localhost API endpoint provided by GPT4all, as per https://docs.gpt4all.io/gpt4all_chat.html#gpt4all-chat-server-mode

I also had to hack all of the constructors of ChatModel() in src/agent/base.py to specify a large request_timeout value (otherwise the local model kept hitting request timeouts).

This seems to be almost working. In the GPT4all server window, I can see the requests coming in, but the agents are not generating responses or making progress. (I tried both the MPT-7b-Chat and MPT-7b-Instruct models).

I think getting this working would be a very fun use case of this tool, as it does not require OpenAI in the loop. I would be quite satisfied even if this means it takes quite a while to wait for the scenario to run.

Is this some limitation of the API exposed by GPT4all? I can provide a branch with my hacks, if that helps make progress. The GPT4all project is quite active; if we can point out why their API is incomplete, they will probably patch it for us.

[dflatline]

gpt4all.patch

This is the patch I used to point GPTeam at GPT4all. Note that you also have to enable the openai endpoint specifically in GPT4all, in the Settings->Application->Enable Web Server.

[rafacost]

I tried to apply the patch but no cigar...
In my case, there is no response or action just after GPTeam GETS of events, on 6469, logs on 60264, and world on 60266.
I am unsure if the request has even been made to GPT4all API.
I think that might be possible to implement support to GPT4all using the python library to interact with langchain.
This link looks promising I think I will give it a try. If I manage to get some results I send a pull request.
Any luck?

[dflatline]

I see the requests in the server window of gpt4all, but it never generates a response. I suspect we might be exceeding the context window of the models it has.

There is also the problem with the smaller models not responding in JSON, apparently: #75 (comment)

So this might be a long road. I think the prompt might need to be trimmed, and possibly also not ask the model to respond in JSON, but in some kind of well-formed dialog format?

That, or we need to wait for bigger version of MPT (since it has large context windows, but is currently only available as a 7B model that probably also can't do JSON well enough).

P.S. I also tried using llama-cpp-python, but it died because gpt4all sends "null" as a max_tokens field in the JSON request. That is this bug over there: abetlen/llama-cpp-python#187