implement prompt template for chat completion

[ehartford]

Is your feature request related to a problem? Please describe.
When generating chat completion, it is hard-coded to generate a non-standard prompt template that looks something like:

### User: <blabla>
### Assistant: <blabla>

system message is currently ignored.

llama-cpp-python/llama_cpp/llama.py

Line 1578 in 255d653

f'### {"Human" if message["role"] == "user" else "Assistant"}:{message["content"]}'

This mostly works for most models. But it's not correct.

Describe the solution you'd like

1. add a set of built-in prompt templates user can specify at inference time ["vicuna","alpaca","chatml","llama2-chat","oasst"] at minimum
2. recommend copying design from ooba's instruction templates or fastchat's conversation
3. add ability to pass a template string for other nonstandard formats (such as the one currently implemented in llama-cpp-python).

Describe alternatives you've considered
modifying llama-cpp-python to hard code it to llama2-chat format, not a great solution.

Additional context

[ehartford]

This is a hefty task, with architecture / design elements to do it in a clean way. I am busy to take it on myself right now, but in a couple weeks I can try if nobody else has done it.

[abetlen]

Hey @ehartford this is actually something I've had in the backlog and just started last night in #711

My plan is to have those format identifiers and also provide a generic class (?) that users can extend to provide a custom chat template. The challenge is that it's not just the prompt that has to be modified but also stop sequences, grammar (in the case of open ai style function calling chats), and a few more things I probably haven't thought about but I think this is do-able.

Thank you for the resources btw!

[ehartford]

Awesome thanks! I think it can be done without requiring the user to write any code, using a clever template system, as is implement by ooba and fastchat.

[lexin4ever]

Like workaround. I use /v1/completions method (create_completion function in llama.py instead of create_chat_completion), which allow me to setup any prompt. And I can format message as I want and pass it as string prompt param.

[ehartford]

true,
but that would require rewriting chatbot-ui to use /completions instead of /chat/completions (or using a reverse proxy to do that)

[ehartford]

then, my solution is that I will make a proxy that receives calls to /chat/completions, and rewrites them into a call into llama-cpp-python's /completions endpoint, in order to inject the proper prompt format.

[teleprint-me]

I just ran through some rough drafts with GPT.

---

Proposal for Advanced Customizable Prompts in Chat Completions

Problem Statement

The existing implementation for chat completions uses hard-coded prompts, constraining customization and flexibility. This limitation becomes evident when adapting the code for specific projects or applications that require unique prompt styles or formats.

    PROMPT = chat_history + "### Assistant:"
    PROMPT_STOP = ["### Assistant:", "### Human:"]

Proposed Solution

I propose two new optional parameters, prompt and prompt_stop, to the create_chat_completion method. These will allow users to specify custom prompt and stop token sequences.

    def create_chat_completion(
        # ...existing parameters...
        prompt: Optional[str] = None,
        prompt_stop: Optional[List[str]] = None,
    ):
        # ...
        PROMPT = chat_history + (prompt if prompt else "### Assistant:")
        PROMPT_STOP = prompt_stop if prompt_stop else ["### Assistant:", "### Human:"]
        # ...

Benefits

1. Enhanced Flexibility: These changes offer users a high level of customization for prompt structures.
2. Wider Applicability: Extending prompt customization increases the adaptability of the code to different use-cases.
3. Ease of Use: The optional parameters maintain the cleanliness of the API while offering more freedom to users.

Backward Compatibility

The proposal maintains backward compatibility since both new parameters are optional and will use existing hard-coded values as defaults.

Suggested Defaults

In absence of custom prompts, the system could default to prompts styled after Llama-2's structure as a sane default:

B_INST, E_INST = "[INST]", "[/INST]"
B_SYS, E_SYS = "<<SYS>>\n", "\n<</SYS>>\n\n"
DEFAULT_SYSTEM_PROMPT = """You are a helpful assistant."""

Practical Examples

    my_custom_prompt = ">>> Custom Assistant:"
    my_custom_stop = [">>> Custom Assistant:", ">>> Custom User:"]
    create_chat_completion(
        messages=...,
        prompt=my_custom_prompt,
        prompt_stop=my_custom_stop,
        # ...other parameters...
    )

Related Works

This proposal aims to integrate well with ongoing work in the configurable-chat-templates branch and issues like #711, focusing on handling bos and eos tokens.

---

I'm not sure if this fits well with what you guys had in mind. Let me know either way. I had the same idea though.

[teleprint-me]

I was looking Open Interpreter and the source code was using litellm. So, I figured I'd take a peek at it and vllm.

I checked out the docs for litellm templates and they have a fairly nice structure for prefixing and postfixing.

# Create your own custom prompt template works 
litellm.register_prompt_template(
        model="togethercomputer/LLaMA-2-7B-32K",
        roles={
            "system": {
                "pre_message": "[INST] <<SYS>>\n",
                "post_message": "\n<</SYS>>\n [/INST]\n"
            },
            "user": {
                "pre_message": "[INST] ",
                "post_message": " [/INST]\n"
            }, 
            "assistant": {
                "post_message": "\n"
            }
        }
    )

def test_huggingface_custom_model():
    model = "huggingface/togethercomputer/LLaMA-2-7B-32K"
    response = completion(model=model, messages=messages, api_base="https://ecd4sb5n09bo4ei2.us-east-1.aws.endpoints.huggingface.cloud")
    print(response['choices'][0]['message']['content'])
    return response

test_huggingface_custom_model()

Found it pretty interesting because you can feed in the structure as a dict and then grab the values by the keys.

I ran through it with GPT again and this is what it came up with as a proof-of-concept.

---

Revised Proposal for Role-Based Customizable Prompts in Chat Completions

Problem Statement

The current chat completions implementation relies on hard-coded prompts, limiting customization and flexibility. This is a bottleneck when adapting the code to specialized projects requiring unique role-based prompt styles or formats.

Proposed Solution

Replace existing prompt and prompt_stop with a single role_templates parameter to the create_chat_completion method. This will offer users the capability to specify custom role-based formatting for different parts of the conversation.

def create_chat_completion(
    # ...existing parameters...
    role_templates: Optional[Dict[str, Dict[str, str]]] = None
):
    # ...existing code...

Benefits

1. Enhanced Flexibility: Users gain high levels of customization for role-based prompt structures.
2. Wider Applicability: The new architecture is adaptable to various chat roles and use-cases.
3. Ease of Use: Replacing multiple parameters with a single role_templates parameter streamlines the API.

Backward Compatibility

This change maintains backward compatibility since the role_templates parameter is optional and defaults to existing hard-coded values if not provided.

Suggested Defaults

A reasonable default could mirror Llama-2's prompt structure:

DEFAULT_ROLE_TEMPLATES = {
    "system": {
        "pre_message": "[INST] <<SYS>>\n",
        "post_message": "\n<</SYS>>\n [/INST]\n"
    },
    "user": {
        "pre_message": "[INST]",
        "post_message": " [/INST]\n"
    },
    "assistant": {
        "post_message": "\n"
    }
}

Related Works

• Inspired by role-based prompt templates in litellm.
• Could also integrate with custom roles specific to llama-cpp-python.

---

I know it won't be that simple after reviewing the code.

Just wanted to share. Maybe it would inspire something.

[NightMachinery]

What models actually use the current chat prompt template?

It seems most models use Alpaca's format:

<System prompt/Character Card>

### Instruction:
Your instruction or question here.
For roleplay purposes, I suggest the following - Write <CHAR NAME>'s next reply in a chat between <YOUR NAME> and <CHAR NAME>. Write a single reply only.

### Response:

[teleprint-me]

@NightMachinery

It depends on the dataset and how it's trained and/or finetuned.

The format varies from model to model, but the 2 most popular formats are usually

"### Instruction:" and "### Assistant:"

and

"### Human:" and "Assistant:"

Sometimes it's "### Human:" and "### Bot:"

Open Assistant uses a mixture depending on version and dataset, "prompter:", or "human:", and "assistant:".

Some models are more complex than others, e.g. it's system prompt, input, instruction, and then response.

There's no fixed, or commonly accepted, format yet as far as I can tell.

Most chat models follow system, user, assistant, or some variation. Whether there are tokens that are used to denote which is which depends.

[ehartford]

The closest thing to standard is ChatML. And it's not widely accepted.

I've adopted it, and open assistant has adopted it. Vicuna and wizardLM haven't.

Hopefully a consensus emerges in the next year.

[teleprint-me]

@ehartford

I'm for ChatML.

The high-level interface is intuitive and easy to reason about and follow.

The low-level interface is similar to what Meta did with Llama-2's chat interface.

The tokens could probably be simplified though. Maybe the use of something more like markup would be an improvement?

<system>System prompt goes here</system>
<user>User prompt goes here</user>
<assistant>

And just "teach" the model that </tag> is always the stop token for that token sequence.

The the output could be parsed similar to XML/HTML.

I'm still learning, so just take what I'm saying with a friendly grain of salt.

This is something I plan on experimenting with if I get the opportunity to do it in the future.

I agree though, a consensus would be nice.

[ehartford]

What models actually use the current chat prompt template?

It seems most models use Alpaca's format:

<System prompt/Character Card>

### Instruction:
Your instruction or question here.
For roleplay purposes, I suggest the following - Write <CHAR NAME>'s next reply in a chat between <YOUR NAME> and <CHAR NAME>. Write a single reply only.

### Response:

The current scheme implemented in llama-cpp-python doesn't follow a convention I know of.

Please see the links in my original issue for a comprehensive and detailed list of the currently popular prompt templates.

90%+ of use cases will be covered if the following formats are supported:

• Llama-2-chat
• ChatML
• Vicuna
• WizardCoder
• Alpaca
• OpenAssistant

The best source of documentation on these prompt formats is probably the model cards in TheBloke's distributions which are very well researched.

[abetlen]

Hey @ehartford I just merged in the #711 PR which adds a mechanism to specify common chat formats through a chat_format parameter on the Llama class and the server settings.

Currently supports:

• llama-2
• alpaca
• vicuna
• oasst_llama
• openbuddy
• redpajama-incite
• snoozy
• phind
• open-orca

Let me know if that works for you!