Local Language Model Implementation Options: Vicuna and Open-Source Alternatives

[lexis33]

It would be fantastic to have the possibility of running it on local Language Model implementations such as Vicuna or other open-source alternatives.

[cstan969]

I recently set it up to run on my local 7B Vicuna model via a custom Langchain classes for VicunaLLM and VicunaChatModel but the issue is that the responses tend to not return in proper JSON format and the system breaks down. So just letting you know that Vicuna is probably not really good enough to work out of the box without some additional tweaking. I could potentially look into LangChain's ReLLM integration to try and get around this problem though. I'm also looking to tweak my setup to get the 13B Vicuna running with CPU assistance. Just figured I'd share my experience with Vicuna here since I saw your comment.

I'm also brand new to this stuff so I could just have messed something up too.

[lexis33]

I recently set it up to run on my local 7B Vicuna model via a custom Langchain classes for VicunaLLM and VicunaChatModel but the issue is that the responses tend to not return in proper JSON format and the system breaks down. So just letting you know that Vicuna is probably not really good enough to work out of the box without some additional tweaking. I could potentially look into LangChain's ReLLM integration to try and get around this problem though. I'm also looking to tweak my setup to get the 13B Vicuna running with CPU assistance. Just figured I'd share my experience with Vicuna here since I saw your comment.

I'm also brand new to this stuff so I could just have messed something up too.

Well, Thank you. I would really like to see if you get it run and if you can later share the code with maybe video or tutorial it will be great. Thank you!

[joshsny]

Generally I'd say the formatting guidance in the prompts is not strong enough, since some of the prompts get too long.

I reckon it is possible to get a version running with smaller models like Vicuna 7B (albeit very slowly), if some of the prompts are reduced in length, so this would be a great place to start if you are looking to go that route.

[lexis33]

Generally I'd say the formatting guidance in the prompts is not strong enough, since some of the prompts get too long.

I reckon it is possible to get a version running with smaller models like Vicuna 7B (albeit very slowly), if some of the prompts are reduced in length, so this would be a great place to start if you are looking to go that route.

Thanks for the information.

Can you tell how you did it? It's asking for API from open ai. I have little experience with python. I overall use oobabooga mostly.