Post process for AI answer

[wss29]

Hi,
when I ask LLM like open ai it returns an answer for me for example
Question: “Write a hello world program using ts”
“Answer:* “{"answer": "typescript\nconsole.log('Hello, World!');\n"}”
1.my problem is that, is there any solution that can post-process the answer before it returns to the chat-ui,
2.for simple I want the initial answer to be passed to a tool method

 postProcess(arg: string): string {
        return 'this is the final answer' + '\n'+ arg;
    }
}

3. the final answer printed to chat-ui should be
   this is the final answer
   typescript

console.log('Hello, World!');

[wss29]

I have read the article of Theia AI Sneak Preview: Let your agents talk to each other!, I am not sure whether agents can be a workflow as follow
input -> agent1 -> agent2 -> agent3 -> chat-ui-response
when I send request to agent1 in ChatInput , agent1 return response1, response1 as request to agent2, agent2 return response2, response2 as request to agent3, agent3 return response to chat-ui

[planger]

Hi,

there is a few approaches you can use, depending on your exact requirements:

1. The most powerful approach is to overwrite AbstractChatAgent.invoke(request: ChatRequestModelImpl). With that you can control the entire handling of the request end-to-end.
   You can first call super.invoke(request), which will invoke the LLM with the user request and stream the result into the request.response, but before returning from invoke(), you pass the response to your own method (postProcess()) to adjust it. If you remove the original response content from the response.contents array and then call response.addContents() with your adjusted content, this should update the chat UI too, because there is a listener event on addContents(), which will trigger the UI update. This should still stream result to the user as it comes in, but once it is finished, the content will be replaced with your modification.
   You can also first call the LLM yourself in your overwrite of invoke() and adjust the content with your postProcess() method and then add the final result into the response.content. This way, your user won't see any result streaming in, but only see the final result.
2. You can overwrite AbstractStreamParsingChatAgent.addContentsToResponse() or parseContents() and process the incoming response before showing it to the user. Have a look at the implementation to see how to best overwrite these for your purposes.
3. If you only want to impact the UI but keep the actual chat model contents in place, you can also bind a ResponseContentMatcherProvider, with which you can define a start and end regular expression to mark a part of the response. For each response part, you can then register a custom ResponsePartRenderer, which returns a ReactNode for this response part. Of course this can filter / post process the LLM content too when creating the ReactNode.

I am not sure whether agents can be a workflow as follow
input -> agent1 -> agent2 -> agent3 -> chat-ui-response
when I send request to agent1 in ChatInput , agent1 return response1, response1 as request to agent2, agent2 return response2, response2 as request to agent3, agent3 return response to chat-ui

For such use cases, we typically overwrite AbstractChatAgent.invoke(request: ChatRequestModelImpl). In your overwritten method, you can call arbitrary other agents or methods, to e.g. augment the chat request, before handling it in the agent that generates and streams the final answer (agent3 in your example). I'd make agent3 in this case the agent that is originally called in which I overwrite the invoke(), because this way, you can eventually just call return super.invoke() and get a streaming response without writing additional code.

[wss29]

Hi @planger ,

Thank you very much for your patient reply. I just have a little problem regarding the context.

My use case involves two steps:

1. I invoke a service to retrieve some context information.
2. I invoke my agent and pass the context. I hope the LLM references the context when answering my question.

We previously used LLMChain, but now we want to switch to using Theia AI.
Here is an example of using context when answering while referencing the context:

LLMChain has a context argument, but the agent in theia AI does not. I’m unsure where I should place the context string properly. Should I add it to the system prompt before invoking the agent?

[planger]

Hi @wss29,

you just need to add the context (e.g. "Wheather report for Boise, ...") to the messages that are sent to the LLM, and there are several ways to do that in a chat agent.

The way that is closest to the "manual approach" you sketched is to overwrite invoke(request) in your chat agent and run custom code to obtain the additional context and add it to the request model before you call super. As input to your custom code (to take the user request into account), you can take the input parameter request, which contains the user request, or you can even access the entire chat session (request.session). To actually obtain the additional context, your custom code could

• Run arbitrary code (to query a database, vector store, invoke internal Theia APIs or even external APIs, etc.)
• Run an additional custom LLM call (e.g. to classify or summarize the chat session content)
• Resolve a variable via the variable service (variables encapsulate the context retrieval into a separate, reusable component -- probably the cleanest approach)
• Call an entirely separate agent

You can then add the obtained result of the code you ran above (context) to the request in your agent's invoke(request) method and continue calling super.invoke(request).

override async invoke(request: ChatRequestModelImpl): Promise<void> {
    const context = await ...; // call an LLM, resolve a variable, or invoke another agent
    // add context to the request as a separate part (alternatively you can also add entire additional messages)
    const lastRequest = request.session.getRequests().slice(-1)[0];
    lastRequest.message.parts.push({
        kind: 'text',
        promptText: context,
        text: '',
        range: { start: 0, endExclusive: 0 }
    });
    // call super with the extended request model
    return super.invoke(request);
}

Hope this helps!

[wss29]

Hi @planger
I followed your guide and managed to get it working. Here’s a screenshot of the result:

It would be more convenient if we passed the request parameter to getSystemMessageDescription.

Thanks again!

[planger]

Good idea to pass the request into getSystemMessageDescription(). Would you mind opening a PR for that improvement? Thank you!

[wss29]

@planger
When I click 'New pull request'

it opens

I can not opening a PR

[planger]

I think you first need to create a fork and then make the changes in your fork and open a PR from your fork against this main repository. Thank you! 👍