Discussion on EvaluationHarness

[TuanaCelik]

This is the discussion board for EvaluationHarness, an experimental component which allows you to orchestrate evaluation for a pipeline with various parameters and for a number of metrics.

The experimental feature includes the RAGEvaluationHarness designed for standard RAG pipelines.

📚 Full Documentation

🧑‍🍳 Cookbook here

Let us know what you think, and if you have any questions about this feature.

Note: Experimental features live in this repository for a fixed period of time. We don't guarantee that we will continue maintaining experimental features. But if they are successful, stable and if you like it, we will move the feature to the core Haystack package.

PS: The first version of this component was implemented by @shadeMe 🚀

[bilgeyucel]

📢 More about EvaluationHarness

Check out our latest cookbook that showcases how to effectively use EvaluationHarness with its override and comparison features: Evaluating AI with Haystack

[d-kleine]

As a feedback, I have noticed that the metrics returned by score_report() will be returned in a unsorted order, which makes it difficult to compare across multiple runs. I think it would be better if the metrics' order in score_report() would match with the defined order in metrics in RAGEvaluationHarness.

pipeline_eval_harness = RAGEvaluationHarness(
    rag_pipeline=basic_rag,
    rag_components=DefaultRAGArchitecture.GENERATION_WITH_EMBEDDING_RETRIEVAL,
    metrics={
        RAGEvaluationMetric.SEMANTIC_ANSWER_SIMILARITY,
        RAGEvaluationMetric.FAITHFULNESS,
        RAGEvaluationMetric.CONTEXT_RELEVANCE,
    }
)

Also, I think the metric_ prefix in the output is redundant, and therefore can be removed for better readability:

From

                    metrics     score
0  metric_context_relevance  0.266667
1                metric_sas  0.537721
2       metric_faithfulness  0.747778

to

                    metrics     score
0         context_relevance  0.266667
1                       sas  0.537721
2              faithfulness  0.747778

[bilgeyucel]

Great feedback, thanks @d-kleine!