Error in ragas==0.3.3: IndexError (list index out of range) causing answer_relevancy to be NaN

[BaqiF2]

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
When using ragas==0.3.3, I encountered the following error:
ERROR - Exception raised in Job[10]: IndexError(list index out of range)
This resulted in the inability to output answer_relevancy, with the final evaluation results as follows:

{'faithfulness': 0.9348, 'context_precision': 1.0000, 'answer_relevancy': nan, 'context_recall': 0.8333}

After downgrading to ragas==0.3.1, the issue was resolved and the evaluation proceeded normally.

Ragas version: 0.3.3
Python version: 3.11

Code to Reproduce
metrics_list = [
Faithfulness(llm=self.ragas_llm),
ContextPrecision(llm=self.ragas_llm),
AnswerRelevancy(llm=self.ragas_llm, embeddings=self.ragas_embeddings),
ContextRecall(llm=self.ragas_llm)
]

evaluate_result = evaluate(
dataset=datasets,
metrics=metrics_list
)

return evaluate_result

[anistark]

Thanks for reporting @784404847
We're taking a look at it.

For a little more context are you using langchain llm ?

[yusufsyaifudin]

also encounter this error, I use langchain-deepseek, is this langchain specific issues @anistark ?

[anistark]

also encounter this error, I use langchain-deepseek, is this langchain specific issues @anistark ?

Most likely, yes. I could reproduce for langchain llm. Will fix it.