Start conversation on Assist Satellite device

[balloob]

Context

With the introduction of Assist Satellite entities, we now have a representation of devices that allow users to talk with Assist. Some of these devices are also able to start a conversation with a user without the user initiating the conversation via a wake-word or button press.

Proposal

Introduce a new Assist Satellite service start_conversation to allow Home Assistant to start talking to a user via an Assist Satellite device. The conversation will always start by an announcement, and then the user has the ability to respond to it.

Use cases:

• User has left the garage door open. Ask if the user wants to close it
• When a user arrives home, summarize the events since they left home and ask them if they want to listen to their favorite music.

Solution

Add a new service assist_satellite.start_conversation that allows targeting assist_satellite entities that support a new supported feature START_CONVERSATION.

The service will have the following service fields:

• start_message and start_media_id. These fields will work exactly like the announce service: either one has to be provided. If no media ID is provided, use TTS of the selected pipeline to generate media.
• extra_system_prompt. This text field will allow providing extra context for the LLM to be able to understand the response of the user. Example can be: "The entity cover.garage_door has been open for 30 minutes and we asked the user if they want to close it". This context is important if the LLM then sees that the user said the words "Sure".

Exploration

• PR to add start_conversation Experiment: Add start_conversation service to assist satellite core#125934
• PR to add extra_system_prompt Add extra prompt to assist pipeline and conversation core#124743

[balloob]

I was having a conversation today with the voice team about this proposal. The problem is that it will work for LLMs but it will not work for the default built-in Home Assistant conversation agent. Given that it's the default, we need to make sure this works.

ResponseIntent

The idea is to add a new type of intent called the "ResponseIntent". These intents are meant to capture responses and not be executed. A response intent has a fixed set of responses, and it can have different sentences mapped to it.

Take for example the YesNoResponseIntent with 2 possible options yes and no and the sentences that match to that:

• yes: "sure", "yes", "ok", "fine"
• no: "nah", "nope", "nada", "no"

Using ResponseIntent

The user will specify in start_conversation that it wants to use an intent_response and that it wants to store the result in a variable.

The user will have to inspect the assist_satellite.start_conversation response to handle the yes and no responses. It will need to leverage assist_satellite.announce to respond to the user when it is done processing.

An automation, would look like this:

triggers: []
conditions: []
actions:
  - action: assist_satellite.start_conversation
    target:
      entity_id: assist_satellite.home_assistant_voice_21c684
    data:
      message: You left the garage door open. Shall I close it?
      intent_response: YesNo
    response_variable: "response"
  - choose:
      - conditions:
          - condition: template
            value_template: "{{ response.intent.response == \"yes\" }}"
        sequence:
          - variables:
              response: Closed
          - action: cover.close_cover
            target:
              entity_id: cover.garage_door
      - conditions:
          - condition: template
            value_template: "{{ response.intent.response == \"no\" }}"
        sequence:
          - variables:
              response: Doing nothing
    default:
      - variables:
          response: I didn't understand it
  - action: assist_satellite.announce
    metadata: {}
    data:
      message: "{{ response }}"
    target:
      entity_id: assist_satellite.home_assistant_voice_21c684

Constraints

The behavior of the service will be slightly different based on what pipeline is selected for the Assist Satellite.

If the pipeline is leveraging the default Home Assistant conversation agent, the user needs to:

• Field intent_response is required
• Storing the response is required
• extra_system_prompt will throw an error
• The return value contains intent, which is an instance of helpers.intent.Intent

If the pipeline is leveraging a conversation agent that uses LLMs:

• The field extra_system_prompt is required
• Storing responses is optional
• Using intent_response will throw an error
• The return value will contain the conversation_result (instance of conversation.agent_manager.ConversationResult) after the LLM has processed the response.

[balloob]

The thing that bothers me about this design is that it supports different fields and responses based on the selected pipeline of the assist satellite entity.

Now, we could (should) capture certain attributes of the primary pipeline of this entity in supported features or other attributes, or else it will not be possible to write a blueprint that deals with this.

It will also require inspection of an Assist Pipeline by Assist Satellite entity to know how to act.

An alternative could be that we split it into two services. Clear what fields/responses are expected. In that case, we would still need a way to know if an entity supports a specific service or not.

[jlpouffier]

Here is how it could work with our built-in intents and sentences.
(Note: I am pretty sure I am going to miss a part somewhere but I hope you'll get the gist)

1. We define ResponseIntent somewhere and link them to sentences, the same way we do for regular intents.
   That would look like this:

language: "en"
intents:
  HassResponseYes:
    response_intent: true
    data:
      - sentences:
          - Yes
          - Yep
          - Sure
          - Absolutely
          - Please do
          - Of course

2. We define sets of sentences that require a continued conversation, and how to behave based on the response intent matched. For that (dummy) example, let's say that we want confirmation before turning off all lights in the home.
   That could look like this:

language: "en"
intents:
 HassTurnOn:
   data:
     # Turn on all lights in the home
     - sentences:
         - "<turn> <all> <light> on <everywhere>"
         - "<turn> on <all> <light> <everywhere>"
         - "activate <all> <light> <everywhere>"
         - "<turn> <light> on <everywhere>"
         - "<turn> on <light> <everywhere>"
         - "activate <light> <everywhere>"
         - "<turn> <all> <light> on"
         - "<turn> on <all> <light>"
         - "activate <all> <light>"
       slots:
         domain: "light"
       continue_conversation:
         response: "Are you sure?" #This can even be a variable that we define in /response like every other response
         response_intents:
           - id: HassResponseYes:
             action: HassTurnOn
           - id: HassResponseNo
             action: HassAbort

Basically, here I am thinking about saying to Hassil "If you get the HassResponseYes in response, then proceed (Perform the same HassTurnOn intent)

BONUS
Imagine we have a common response intent called HassResponseDuration that contains slots for hours, minutes and seconds.
Something that may look like this:

language: "en"
intents:
  HassResponseDuration:
    response_intent: true
    data:
      - sentences:
          - <timer_duration>
        slots:
          hour: hours
          minute: minutes
          second: seconds

Then we could add missing slots to an intent using the response intents

Like that:

language: "en"
intents:
  HassStartTimer:
    data:
      - sentences:
          - "Start a timer"
        continue_conversation:
          response: "For how long?"
          response_intents:
            - id: HassResponseDuration:
              action: HassStartTimer
              slots:
                hour: hours
                minute: minutes
                second: seconds

Basically, we say "The HassStartTimer intent requires a few slots that these sentences do not provide, so we are going to continue the conversation to fetch them and fulfill the intent"

Pretty sure I am missing dozens of use-cases, but I hope it's somehow useful to keep the ball rolling.

[balloob]

I've talked to some people in the core team. The suggestion was to extract this idea into its own proposal, which I did here: #1150. It will start small and just return the raw value.