AI Workspace agent is slow and inaccurate #14361
Comments
|
I suppose the essence of what I am saying here is that I expect Theia AI, which has several agents as well as an orchestrator between them, to be able to take the smart route to answering an AI chat query, in this particular case being for it to understand that it can answer this faster, cheaper, better with a simple terminal command and to ideally do it (or instruct the user on it at the terminal, but really ideally Theia AI should be able to access the terminal and do it I think [or are there some serious security risks with that?]). (And at the very least if it is still going to answer the question without terminal use, it should answer the question correctly. And ideally also quickly/cheaply.) |
|
Thank you for this report. First of all, if you have any concrete optimizations of prompts or agents in mind, please feel free to suggest contributions. The AI support in the Theia IDE is in an early, experimental state and far from beeing optimized. When I try to extract your report into potential actions to take, I see he following four:
Does this make sense to you? |
|
Yes, sounds good. And thanks for putting together and launching Theia AI!
(And Theia generally.)
…On Wed, Oct 30, 2024, 12:41 Jonas Helming ***@***.***> wrote:
Thank you for this report. First of all, if you have any concrete
optimizations of prompts or agents in mind, please feel free to suggest
contributions. The AI support in the Theia IDE is in an early, experimental
state and far from beeing optimized.
When I try to extract your report into potential actions to take, I see he
following four:
1. Workspace agents does not answer the question correctly => This
could be due to the size of the context window, this should be checked.
2. Enable agents to use the terminal => This is an interesting route,
but potentially pretty dangerous without the user reviewing commands. So I
would rather display them to the user and let the user execute them then
3. Enable a terminal chat agent => We have the terminal assistant
embedded in the terminal view, but currently no chat agent for creating
terminal commands, but we should have one
4. More intelligent orchestrator => The flow you describe makes
perfect sense. This is an optimization of the orchestrator prompt/flow in a
way that it would use the terminal chat agent (not yet existing) for this
example request.
Does this make sense to you?
—
Reply to this email directly, view it on GitHub
<#14361 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAN2PSKBFOORSNXMWO5DQULZ6CZT5AVCNFSM6AAAAABQXGGW5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBWGUZDCNZRG4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
fixed #14361 Signed-off-by: Jonas Helming <jhelming@eclipsesource.com>
Bug Description:
The Theia AI Workspace Agent is slow and inaccurate on a basic 'smoke test', when asked a simple question: "how many files are in this repo?" while open in a workspace containing the theia repo.
Steps to Reproduce:
openai/gpt-4o-mini.https://github.com/eclipse-theia/theiaand open it in theia as a workspace.files = <paste>and counting withlen(files)says there are 3757 entries in this list.Partial view of what the list of files used as intermediary by Workspace Agent looked like (GitHub prevented the entire list due to a 65536 character limit on comment length)
Additional Information
When I asked with the Perplexity API model (
llama-3.1-sonar-small-128k-online), I received the following different and not useful answer (not useful in that it did not execute this procedure and provide me with an answer):My point in mentioning this is that the proper expected result, properly, in my opinion, is that the AI agent would perform some combination of these answers: it should (and has the ability to, it seems) understand that it can run one simple terminal command could answer the question, and it should perform this command (similar to how it performs getWorkspaceFileList, although it is perhaps a bit different as that function is presumably hardcoded in theia?) and return the output of that command.
This is both a convenience matter, in that the answer should be provided near-instantly at the speed of non-AI computation of a simple terminal command (when I ran the suggested
git ls-files | wc -lin the theia terminal it returned instantly — but interestingly with a different number than my python counting of the getWorkspaceFilesList... 3106), but it is also a cost issue: there is no need to send a list of 3757 filenames to the AI model, which the API charges the user for per-token.Another issue, when I enabled all the AI agents, each with
gpt-4o-mini, and I asked the chat (so that it goes initially to the orchestrator, although I did not actually see the orchestrator in action) it still used the workspace agent but now gave a different answer:System info
The text was updated successfully, but these errors were encountered: