Exception raised in Job[2]: RagasOutputParserException(The output parser failed to parse the output including retries.)

[shanchao0906]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question
what is unclear to you? What would you like to know?

Code Examples
This community speaks code. Share your code snippets to help us understand your question better.

Additional context
Anything else you want to share with us?

ragas 版本 0.2.14
llm_model="qwen2.5:32b"

llm = OllamaLLM(base_url = "http://xxx/", model=llm_model)
llm = ChatOllama(base_url = "http://xxx/", model=llm_model,max_tokens=160000)
evaluator_llm = LangchainLLMWrapper(llm,run_config)
#测试是否调用成功
response = llm.invoke("讲一个笑话")
print(response)

embed_model="quentinz/bge-large-zh-v1.5:f32"
embeding = OllamaEmbeddings(base_url = "http://xxxx",model=embed_model/)
wrapped_embeddings = LangchainEmbeddingsWrapper(embeding)

测试生成嵌入向量
text = "This is a test sentence."
embed_vector = embeding.embed_query(text)
以上代码测试没有问题

questions = [
"自动化技术与应用专业的入学基本要求？"
]

correct_answers = [
"中等职业学校毕业、普通高级中学毕业或具备同等学力"
]

samples = []

对每个问题进行处理
for question, correct_answer in zip(questions, correct_answers):
res = session.ask(question, stream=True)

存储 AI 给出的最终答案及检索召回的文本片段

final_answer = ""
retrieved_contexts = []

for ans in res:
final_answer = ans.content
if ans.reference is not None:
for ref in ans.reference:
retrieved_contexts.append(ref['content'])
print("final_answer:",final_answer)
print("retrieved_contexts:",retrieved_contexts)

创建 SingleTurnSample 对象

sample = SingleTurnSample(
user_input=question,
retrieved_contexts=retrieved_contexts,
response=final_answer,
reference=correct_answer
)
samples.append(sample)
print("samples")

生成评估数据集
eval_dataset = EvaluationDataset(samples=samples)
print(eval_dataset)

选择评估指标进行评估
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness, SemanticSimilarity
from ragas import evaluate

metrics = [
LLMContextRecall(llm=evaluator_llm),
FactualCorrectness(llm=evaluator_llm),
Faithfulness(llm=evaluator_llm),
SemanticSimilarity(embeddings=wrapped_embeddings)
]
results = evaluate(dataset=eval_dataset, metrics=metrics)
print(results)

设置显示所有列，打印结果
import pandas as pd

pd.set_option('display.max_columns', None)
print(results.to_pandas())
df = results.to_pandas()
print(df.head())
df.head().to_csv('res.csv', index=False,encoding='utf-8-sig')

Image
评估结果也是NAN 太难受了 排查好几天了 也不知道问题

[shanchao0906]

[shanchao0906]

28<00:09, 9.13s/it]output_string: ```json
{
"statements": [
{
"statement": "根据提供的文档内容，虽然没有直接提到自动化技术与应用专业，但可以参考类似的专业如机电一体化技术的入学基本要求。",
"reason": "输入文本中确实未提及自动化技术与应用专业的具体入学要求，而是提供了两个相关专业（自动化技术和工业过程自动化技术）的具体信息。因此，该陈述是合理的推断。",
"verdict": 1
},
{
"statement": "通常情况下，这类理工科类的职业教育专业会有相似的要求。",
"reason": "输入文本中提到的入学要求确实与理工科类职业教育专业类似，但没有直接证据支持所有相关专业的入学基本要求都相同。因此，该陈述是合理的推测。",
"verdict": 1
},
{
"statement": "对于机电一体化技术（460301）专业的入学基本要求是中等职业学校毕业。",
"reason": "输入文本没有提到机电一体化技术专业具体入学要求。因此，该陈述缺乏直接证据支持。",
"verdict": 0
},
{
"statement": "对于机电一体化技术（460301）专业的入学基本要求是普通高级中学毕业。",
"reason": "输入文本没有提到机电一体化技术专业具体入学要求。因此，该陈述缺乏直接证据支持。",
"verdict": 0
},
{
"statement": "对于机电一体化技术（460301）专业的入学基本要求是具备同等学力。",
"reason": "输入文本没有提到机电一体化技术专业具体入学要求。因此，该陈述缺乏直接证据支持。",
"verdict": 0
},
{
"statement": "因此，自动化技术与应用相关专业的入学基本要求很可能也是面向完成初中教育并获得相应学历证明的学生。",
"reason": "输入文本中提到的两个专业（自动化技术和工业过程自动化技术）都要求中等职业学校毕业、普通高级中学毕业或具备同等学力。该陈述是合理的推测。",
"verdict": 1
},
{
"statement": "自动化技术与应用相关专业的入学基本要求也可能面向具有同等学习能力的个体。",
"reason": "输入文本提到的专业（自动化技术和工业过程自动化技术）都要求具备同等学力。该陈述是合理的推测。",
"verdict": 1
},
{
"statement": "具体的要求可能会因不同的教育机构而有所差异，请以实际招生简章为准。",
"reason": "输入文本中提到的入学基本要求可能因不同学校或机构而异，因此该陈述合理。",
"verdict": 1
}
]
}

注意：在上述JSON输出中，`"statement"`字段中的某些内容被错误地拆分了。正确的表述应为：

- `"statement": "对于机电一体化技术（460301）专业的入学基本要求是中等职业学校毕业。"`
- `"statement": "对于机电一体化技术（460301）专业的入学基本要求是普通高级中学毕业。"`
- `"statement": "对于机电一体化技术（460301）专业的入学基本要求是具备同等学力。"`

这些陈述在原始输入中被正确地表述，但在输出中的某些地方出现了错误的拆分。
output_string: It seems like the input text is a bit disorganized and contains some errors. However, I can summarize the key points from it:

The context describes three vocational education programs related to automation technology and application:
1. Automation Technology & Application (专业代码 2605)
   - Basic Admission Requirements: Graduates of secondary vocational schools or high school graduates with equivalent academic ability.
   - Duration: Four years
   - Goal: Cultivate students who can practice core socialist values, inherit skill civilization, develop in morality, intelligence, physical fitness, aesthetics and labor skills. They should have a higher level of scientific culture, good humanistic action abilities.

2. Electrical Automation Technology (专业代码 460306)
   - Basic Admission Requirements: Graduates from secondary vocational schools or high school graduates with equivalent academic ability.
   - Duration: Three years
   - Goal: Cultivate students who can practice core socialist values, inherit skill civilization, develop in morality, intelligence, physical fitness, aesthetics and labor skills. They should have a certain level of scientific culture, good humanistic literacy, science literacy, digital literacy, professional ethics, innovation awareness, love for job and dedication spirit, meticulous craftsmanship spirit, strong employment and entrepreneurship abilities, sustainable development capabilities.

3. Industrial Automation Instrument Technology (专业代码 460308)
   - Basic Admission Requirements: Graduates from secondary vocational schools or high school graduates with equivalent academic ability.
   - Duration: Three years
   - Goal: Cultivate students who can practice core socialist values, inherit skill civilization, develop in morality, intelligence, physical fitness, aesthetics and labor skills. They should have a certain level of scientific culture, good humanistic literacy, science literacy, digital literacy, professional ethics, innovation awareness, love for job and dedication spirit, meticulous craftsmanship spirit, strong employment and entrepreneurship abilities.

The text also mentions that the specific admission requirements may vary among different educational institutions and advises to refer to the actual enrollment brochure. 

[shanchao0906]

@dosu 解答一下原因谢谢

[dosubot]

The RagasOutputParserException you're encountering is triggered when the output parser fails to parse the output, even after retries. This can happen if the output string does not meet the constraints specified in the prompt and cannot be corrected within the allowed number of retries. The framework attempts to fix the output string using a prompt and retries parsing it, but if it remains invalid after the specified retries, the exception is raised ^[1][2].

To address this issue, ensure that the model's output is JSON-parsable and adheres to the expected format. You might need to review the specific outputs causing the exception and adjust the model or prompt to produce valid JSON responses. Additionally, check if the output is being split incorrectly, as mentioned in your JSON output example, and ensure that the statements are correctly formatted before parsing.

If you are also encountering NaN values in your evaluation results, this could be due to JSON parsing issues or non-ideal cases for scoring, such as responses that are not suitable for evaluation. Make sure the model's output is structured correctly and consider upgrading to Ragas version 0.2.9 or later, where some NaN issues have been addressed ^[3].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[674717172]

同样问题，解决了吗兄弟