Skip to content

Commit 4026f4f

Browse files
authored
fix: answer_relevancy scoring logic to prevent false zero (#2327)
## Issue Link / Problem Description <!-- Link to related issue or describe the problem this PR solves --> - Fixes #2326 ## Changes Made <!-- Describe what you changed and why --> - fixed answer_relevancy scoring logic to prevent false zero scores for valid answers
1 parent 8b943a7 commit 4026f4f

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

src/ragas/metrics/_answer_relevance.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -116,15 +116,15 @@ def _calculate_score(
116116
) -> float:
117117
question = row["user_input"]
118118
gen_questions = [answer.question for answer in answers]
119-
committal = np.any([answer.noncommittal for answer in answers])
119+
all_noncommittal = np.all([answer.noncommittal for answer in answers])
120120
if all(q == "" for q in gen_questions):
121121
logger.warning(
122122
"Invalid JSON response. Expected dictionary with key 'question'"
123123
)
124124
score = np.nan
125125
else:
126126
cosine_sim = self.calculate_similarity(question, gen_questions)
127-
score = cosine_sim.mean() * int(not committal)
127+
score = cosine_sim.mean() * int(not all_noncommittal)
128128

129129
return score
130130

0 commit comments

Comments
 (0)