diff --git a/README.md b/README.md index a0e0ea2e01262..24482a223a8a9 100644 --- a/README.md +++ b/README.md @@ -555,6 +555,10 @@ Here is an example of a few-shot interaction, invoked with the command # custom arguments using a 13B model ./main -m ./models/13B/ggml-model-q4_0.bin -n 256 --repeat_penalty 1.0 --color -i -r "User:" -f prompts/chat-with-bob.txt + +# chat with LlaMa-2 chat models (handles special system and instruction tokens) +# second argument is system prompt and third one is first user prompt +./examples/chat-llama-2.sh models/llama-2-13b-chat.ggmlv3.q4_0.bin ./prompts/pirate.txt "Hello there" ``` Note the use of `--color` to distinguish between user input and generated text. Other parameters are explained in more detail in the [README](examples/main/README.md) for the `main` example program. diff --git a/examples/chat-llama-2.sh b/examples/chat-llama-2.sh new file mode 100755 index 0000000000000..178364ec4eaa9 --- /dev/null +++ b/examples/chat-llama-2.sh @@ -0,0 +1,11 @@ +#!/bin/bash + +# The script should be launched like ./chat.sh models/llama-2-13b-chat.ggmlv3.q4_0.bin system_prompts/translation.txt Hello + +# Load system prompt +SYSTEM_PROMPT=$(cat $2) + +# Execute model +./main -m $1 -c 4096 -n -1 --in-prefix-bos --in-prefix ' [INST] ' --in-suffix ' [/INST]' -ngl 40 -i \ + -p "[INST] <>\n$SYSTEM_PROMPT\n<>\n\n$3 [/INST]" + diff --git a/prompts/pirate.txt b/prompts/pirate.txt new file mode 100644 index 0000000000000..1c5500a532580 --- /dev/null +++ b/prompts/pirate.txt @@ -0,0 +1 @@ +You are a helpful assitant that speaks pirate \ No newline at end of file