add action input as parameters for tool execution in conversational agent

[jngz-es]

Description

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#3134

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[mingshl]

tests passed but failed in upload, not related to this code change. Approved.

Run actions/upload-artifact@v4
/usr/bin/docker exec  a6c6ef6ad38d2e9993d03ba2bdc50f2146f892a4a109fc9fecaf2c66802943f0 sh -c "cat /etc/*release | grep ^ID"
/__e/node[20](https://github.com/opensearch-project/ml-commons/actions/runs/11670434010/job/32954019963?pr=3200#step:8:21)/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
``'

[reuschling]

The changes for AgentUtils look fine, but is AgentUtils used for conversational agents? The tool parameters are build inside MLConversationalFlowAgentRunner.getToolExecuteParams(MLToolSpec toolSpec, Map<String, String> params), i.e. here. There is no AgentUtils invocation. The only invocation of AgentUtils.‎‎constructToolParams I found is inside MLChatAgentRunner.

I showed a code proposal for getToolExecuteParams at #2977 (comment), the only difference is that there is no dedicated actionInput parameter, the actionInput is the "input" entry so far, which has to be temporarily stored inside a local variable.

I would also highly recommend to add the new "action_input" parameter to flow agents (i.e. MLFlowAgentRunner.getToolExecuteParams) also. There is no use of AgentUtils too. Of course there is the possibility with parameters.previous_tool_name.output, but tool specifications should act the same independent where they should be used, whether inside flow or conversational agents.

[jngz-es]

The changes for AgentUtils look fine, but is AgentUtils used for conversational agents? The tool parameters are build inside MLConversationalFlowAgentRunner.getToolExecuteParams(MLToolSpec toolSpec, Map<String, String> params), i.e. here. There is no AgentUtils invocation. The only invocation of AgentUtils.‎‎constructToolParams I found is inside MLChatAgentRunner.

I showed a code proposal for getToolExecuteParams at #2977 (comment), the only difference is that there is no dedicated actionInput parameter, the actionInput is the "input" entry so far, which has to be temporarily stored inside a local variable.

I would also highly recommend to add the new "action_input" parameter to flow agents (i.e. MLFlowAgentRunner.getToolExecuteParams) also. There is no use of AgentUtils too. Of course there is the possibility with parameters.previous_tool_name.output, but tool specifications should act the same independent where they should be used, whether inside flow or conversational agents.

Hi @reuschling , thanks for the comments. Yeah, you are right. The changes is only for conversational agents. As you also mentioned, the parameters.previous_tool_name.output is designed as action input for flow agents which is a sequence of tools. I don't see a use case of flow agent where the parameters.previous_tool_name.output could not meet the requirement but the action input did.

[reuschling]

Hi @reuschling , thanks for the comments. Yeah, you are right. The changes is only for conversational agents.

Sorry, but I think you misunderstood me. Currently this PR makes NO change to conversational agents. It changes AgentUtils which is ONLY invoked inside MLChatAgentRunner. For conversational agents, you have to modify MLConversationalFlowAgentRunner.

As you also mentioned, the parameters.previous_tool_name.output is designed as action input for flow agents which is a sequence of tools. I don't see a use case of flow agent where the parameters.previous_tool_name.output could not meet the requirement but the action input did.

Exactly. But when someone wants to use the same tool inside a conversational agent and inside a flow agent, he/she has to change the tool definition from parameters.previous_tool_name.output to action input or vice versa. Why giving the same thing different names? It's not for more functionality, but for logic and design purposes.

[ylwu-amzn]

@reuschling Thanks for reviewing this PR. From #2918 (comment), I think you know "MLChatAgentRunner uses the AgentUtils method AgentUtils.constructToolParams for generating the params for a tool."

The name seems confusing, but actually MLChatAgentRunner is for conversational agent (code link).
Refer to this tutorial for differences between different agent types.

Test

I have tested this PR with Bedrock anthropic.claude-instant-v1 model.

1. You should create a test_population_data index , which has a text field population_description first. Refer to this tutorial for creating this index.
2. Need to configure config of SearchIndexTool. Add a static input template with placeholder llm_generated_action_input. The placeholder llm_generated_action_input will be substituted with input generated by LLM.

POST _plugins/_ml/agents/_register
{
    "name": "Test Agent",
    "type": "conversational",
    "description": "Simple agent to test the agent framework",
    "llm": {
        "model_id": "<your LLM model id>",
        "parameters": {
            "max_iteration": 5,
            "stop_when_no_tool_found": true,
            "disable_trace": false
        }
    },
    "memory": {
        "type": "conversation_index"
    },
    "app_type": "chat_with_rag",
    "tools": [
        {
            "type": "SearchIndexTool",
            "description": "A tool to search opensearch index with natural language question. If you don't know answer for some question, you should always try to search data with this tool. Action Input: <natural language question>",
            "include_output_in_agent_response": true,
            "config": {
                "input": "{\"index\": \"test_population_data\", \"query\": {\"query\":{\"match\":{\"population_description\":\"${parameters.llm_generated_action_input}\"}}} }"
            }
        }
    ]
}

Test agent with

{
  "parameters": {
    "question": "what's the population increase of Seattle from 2021 to 2023?"
  }
}

Feel free to test. BTW, you can find @jngz-es and me in the public OpenSearch ml Slack channel https://join.slack.com/t/opensearch/shared_invite/zt-2r5scz3ty-5SMPhqJE_Lk2HqC6ex4mWg, welcome to join. I'm ok to jump to a call if that's easier to explain details with a demo.

[ylwu-amzn]

Maybe no need to mention action explicitly considering REST API uses tool. User may feel confused about tool and action. How about just llm_generated_input ?

[pyek-bot]

can also consider using constant for this string since it is reused in the tests

[reuschling]

@reuschling Thanks for reviewing this PR. From #2918 (comment), I think you know "MLChatAgentRunner uses the AgentUtils method AgentUtils.constructToolParams for generating the params for a tool."

The name seems confusing, but actually MLChatAgentRunner is for conversational agent (code link).

Thanks a lot @ylwu-amzn for clarification with the code link. Yes the name confused me, sorry about that @jngz-es . In this case I'm also fine with the changes :)