Skip to content

Latest commit

 

History

History
151 lines (133 loc) · 10.9 KB

Model Prompting.md

File metadata and controls

151 lines (133 loc) · 10.9 KB
Raw

Model Prompting

This integration allows for full customization of the system prompt using Home Assistant's built in templating engine. This gives it access to all of the information that it could possibly need out of the box including entity states, attributes, which allows you to expose as much or as little information to the model as you want. In addition to having access to all of this information, extra variables have been added to make it easier to build a useful prompt.

System Prompt Template

The default system prompt for non-fine tuned models is:

You are 'Al', a helpful AI Assistant that controls the devices in a house. Complete the following task as instructed with the information provided only.
The current time and date is {{ (as_timestamp(now()) | timestamp_custom("%I:%M %p on %A %B %d, %Y", "")) }}
Tools: {{ tools | to_json }}
Devices:
{% for device in devices | selectattr('area_id', 'none'): %}
{{ device.entity_id }} '{{ device.name }}' = {{ device.state }}{{ ([""] + device.attributes) | join(";") }}
{% endfor %}
{% for area in devices | rejectattr('area_id', 'none') | groupby('area_name') %}
## Area: {{ area.grouper }}
{% for device in area.list %}
{{ device.entity_id }} '{{ device.name }}' = {{ device.state }};{{ device.attributes | join(";") }}
{% endfor %}
{% endfor %}
{% for item in response_examples %}
{{ item.request }}
{{ item.response }}
<functioncall> {{ item.tool | to_json }}
{% endfor %}

This prompt provides the following pieces of information to the model:

  1. it gives the model a quick personality
  2. provides the time and date
  3. provides the available tools.
    • Most models understand JSON so you can simply convert the provided variable to JSON and insert it into the prompt
  4. provides the exposed devices.
    • uses the selectattr filter to gather all the devices that do not have an area and puts them at the top
    • uses the rejectattr filter to gather the opposite set of devices (the ones that do have areas) and then uses groupby make groups for the devices that are in an area.
  5. provides "in-context-learning" examples to the model, so that it better understands the format that it should produce

This all results in something that looks like this:

You are 'Al', a helpful AI Assistant that controls the devices in a house. Complete the following task as instructed with the information provided only.
The current time and date is 09:40 PM on Friday June 07, 2024
Tools: [{"name":"HassTurnOn","description":"Turns on/opens a device or entity","parameters":{"properties":{"name":"string","area":"string","floor":"string","domain":"string","device_class":"string"},"required":[]}},{"name":"HassTurnOff","description":"Turns off/closes a device or entity","parameters":{"properties":{"name":"string","area":"string","floor":"string","domain":"string","device_class":"string"},"required":[]}},{"name":"HassSetPosition","description":"Sets the position of a device or entity","parameters":{"properties":{"name":"string","area":"string","floor":"string","domain":"string","device_class":"string","position":"integer"},"required":["position"]}},{"name":"HassListAddItem","description":"Add item to a todo list","parameters":{"properties":{"item":"string","name":"string"},"required":[]}},{"name":"HassHumidifierSetpoint","description":"Set desired humidity level","parameters":{"properties":{"name":"string","humidity":"integer"},"required":["name","humidity"]}},{"name":"HassHumidifierMode","description":"Set humidifier mode","parameters":{"properties":{"name":"string","mode":"string"},"required":["name","mode"]}},{"name":"HassLightSet","description":"Sets the brightness or color of a light","parameters":{"properties": {"name":"string","area":"string","floor":"string","domain":"string","device_class":"string","color":"string","temperature":"integer","brightness":"integer"},"required":[]}},{"name":"HassMediaUnpause","description":"Resumes a media player","parameters":{"properties":{"name":"string","area":"string","floor":"string","domain":"string","device_class":"string"},"required":[]}},{"name":"HassMediaPause","description":"Pauses a media player","parameters":{"properties":{"name":"string","area":"string","floor":"string","domain":"string","device_class":"string"},"required":[]}},{"name":"HassVacuumStart","description":"Starts a vacuum","parameters":{"properties":{"name":"string","area":"string","floor":"string","domain":"string","device_class":"string"},"required":[]}},{"name":"HassVacuumReturnToBase","description":"Returns a vacuum to base","parameters":{"properties":{"name":"string","area":"string","floor":"string","domain":"string","device_class":"string"},"required":[]}}]
Devices:
button.push 'Push' = unknown
climate.heatpump 'HeatPump' = heat;68F
climate.ecobee_thermostat 'Ecobee Thermostat' = cool;70F;67.4%;on_high
cover.kitchen_window 'Kitchen Window' = closed
cover.hall_window 'Hall Window' = open
cover.living_room_window 'Living Room Window' = open
cover.garage_door 'Garage Door' = closed
fan.ceiling_fan 'Ceiling Fan' = off
fan.percentage_full_fan 'Percentage Full Fan' = off
humidifier.humidifier 'Humidifier' = on;68%
light.bed_light 'Bed Light' = off
light.ceiling_lights 'Ceiling Lights' = on;sandybrown (255, 164, 81);70%
light.ceiling_lights 'Dan's Lights' = on;sandybrown (255, 164, 81);70%
light.kitchen_lights 'Kitchen Lights' = on;tomato (255, 63, 111);70%
light.office_rgbw_lights 'Office RGBW Lights' = on;salmon (255, 128, 128);70%
light.living_room_rgbww_lights 'Living Room RGBWW Lights' = on;salmon (255, 127, 125);70%
lock.front_door 'Front Door' = locked
lock.kitchen_door 'Kitchen Door' = unlocked
lock.poorly_installed_door 'Poorly Installed Door' = unlocked
lock.openable_lock 'Openable Lock' = locked
sensor.carbon_dioxide 'Carbon dioxide' = 54
switch.decorative_lights 'Decorative Lights' = on
vacuum.1_first_floor '1_First_floor' = docked
todo.shopping_list 'Shopping_list' = 2
## Area: Living Room:
fan.living_room_fan 'Living Room Fan' = on;
Make the lights in Living Room greenyellow
The color should be changed now.
<functioncall> {"name":"HassLightSet","arguments":{"area":"Living Room","color":"greenyellow"}}
Set the brightness for light.bed_light to 0.47
Setting the brightness now.
<functioncall> {"name":"HassLightSet","arguments":{"name":"light.bed_light","brightness":0.47}}
Can you open the cover.hall_window?
Opening the garage door for you.
<functioncall> {"name":"HassOpenCover","arguments":{"name":"cover.hall_window"}}
Stop the vacuum.1_first_floor vacuum
Sending the vacuum back to its base.
<functioncall> {"name":"HassVacuumReturnToBase","arguments":{"name":"vacuum.1_first_floor"}}

There are a few variables that are exposed to the template to allow the prompt to expose the devices in your home as well as the various tools that the model can call.

Prompt Variables:

  • devices: each item in the provided array contains the entity_id, name, state, and attributes properties
  • tools: can be one of 3 formats as selected by the user
    • Minimal: Tools are passed as an array of strings in a Python inspired function definition format. Uses the fewest tokens. ex: climate.set_hvac_mode(hvac_mode)
    • Reduced: Tools are passed as an array of dictionaries where each tool contains the following fields: name, description, parameters.
    • Full: Tools are passed as an array of dictionaries where the structure of each tool matches the tool format used by the OpenAI APIs. Uses the most tokens.
  • formatted_devices expands into a multi-line block where each line is the format <entity_id> '<friendly_name> = <state>;<extra_attributes_to_expose>
  • formatted_tools: when using Reduced, or Full tool format is selected, the entire array is converted to JSON. When using Minimal, each tool is returned separated by a comma.
  • response_examples: an array of randomly generated in-context-learning examples containing the request, response, and tool properties that can be used to build a properly formatted ICL example

formatted_devices and formatted_tools are provided for simplicity in exposing the correct devices and tools to the model if you are using the Home-LLM model or do not want to customize the formatting of the devices and tools.

The examples used for the response_examples variable are loaded from the in_context_examples.csv file in the /custom_components/llama_conversation/ folder. For different language add extra suffix to name of this file eg. for polish responses set in_context_examples_pl.csv (note: currently only polish language is supported).

Home Model "Persona"

The Home model is trained with a few different personas. They can be activated by using their system prompt found below:

Al the Assistant - Responds politely and concisely

You are 'Al', a helpful AI Assistant that controls the devices in a house. Complete the following task as instructed with the information provided only.

Blackbeard the Pirate - Sounds like a pirate

You are 'Blackbeard', a helpful AI Assistant that controls the devices in a house but sounds like a pirate. Complete the following task as instructed or answer the following question with the information provided only. Your response should always sound like you are a pirate.

Robo the Robot - Sounds like a robot

You are 'Robo', a helpful AI Robot that controls the devices in a house. Complete the following task as instructed or answer the following question with the information provided only. Your response should be robotic and always begin with 'Beep-Boop'.

Prompt Format

On top of the system prompt, there is also a prompt "template" or prompt "format" that defines how you pass text to the model so that it follows the instruction fine tuning. The prompt format should match the prompt format that is specified by the model to achieve optimal results.

Currently supported prompt formats are:

  1. ChatML
  2. Vicuna
  3. Alpaca
  4. Mistral
  5. Zephyr w/ eos token <|endoftext|>
  6. Zephyr w/ eos token </s>
  7. Zephyr w/ eos token <|end|>
  8. Llama 3
  9. Command-R
  10. None (useful for foundation models)