Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Port evaluation harness from haystack-experimental #8595

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

shadeMe
Copy link
Contributor

@shadeMe shadeMe commented Nov 29, 2024

Proposed Changes:

Port the evaluation harness from the experimental package.

How did you test it?

Unit tests

Notes for the reviewer

The source feature branch doesn't originate from a fork, so you can directly push to branch if need be.

We should ensure that we have the usage docs for the harness and its related classes ready so that both the code and the docs are merged at the same time, i.e., during the same release.

@dfokina Feel free to review the docstrings as well.

Checklist

@shadeMe shadeMe requested a review from dfokina November 29, 2024 14:23
@github-actions github-actions bot added topic:tests type:documentation Improvements on the docs labels Nov 29, 2024
@coveralls
Copy link
Collaborator

coveralls commented Nov 29, 2024

Pull Request Test Coverage Report for Build 12200323610

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 33 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.4%) to 90.707%

Files with Coverage Reduction New Missed Lines %
components/retrievers/sentence_window_retriever.py 1 98.08%
components/preprocessors/nltk_document_splitter.py 3 97.35%
components/converters/pypdf.py 6 93.55%
components/converters/azure.py 23 89.38%
Totals Coverage Status
Change from base Build 12086061543: 0.4%
Covered Lines: 8453
Relevant Lines: 9319

💛 - Coveralls

@julian-risch julian-risch self-assigned this Dec 6, 2024
@julian-risch julian-risch marked this pull request as ready for review December 6, 2024 14:14
@julian-risch julian-risch requested review from a team as code owners December 6, 2024 14:14
@julian-risch julian-risch requested review from vblagoje and removed request for a team December 6, 2024 14:14
@dfokina dfokina added this to the 2.9.0 milestone Dec 18, 2024
@bilgeyucel
Copy link
Contributor

bilgeyucel commented Dec 25, 2024

Can we add the DocumentNDCGEvaluator to the supported metrics as we port it?
Another small feature request from the community: deepset-ai/haystack-experimental#74 (reply in thread)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic:tests type:documentation Improvements on the docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants