Implementation of Llama for the Leolani platform. It requires a Llama server to send requests to.
https://python.langchain.com/docs/integrations/llms/llamacpp/
pip install llama-cpp-python[server]==0.2.62 pip install openai
Download the Llama file from:
pip install sse_starlette pip install starlette_context pip install pydantic_settings
#with CPU only python -m llama_cpp.server --host 0.0.0.0 --model ./models/Meta-Llama-3-8B-Instruct.Q2_K.gguf --n_ctx 2048
#If you have a NVidia GPU python -m llama_cpp.server --host 0.0.0.0 --model ./models/Meta-Llama-3-8B-Instruct.Q2_K.gguf --n_ctx 2048 --n_gpu_layers 28
This repository is a component of the Leolani framework. For usage of the component within the framework see the instructions there.
Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Distributed under the MIT License. See LICENSE for more information.