Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support document summarization evaluation with microservice. #34

Merged
merged 4 commits into from
Jun 26, 2024

Conversation

lkk12014402
Copy link
Collaborator

@lkk12014402 lkk12014402 commented Jun 17, 2024

Description

add document summarization metrics to evaluate DocSum microservice/megaservice

@lkk12014402
Copy link
Collaborator Author

lkk12014402 commented Jun 17, 2024

Usage

basic Rouge metrics

only evaluation

from evals.metrics.summarization import SummarizationMetric

sum_eval = SummarizationMetric()

sum_eval.rouge_scores(eval_summary, ref_summary)
print(sum_eval.average_score)

generation & evaluation

from evals.metrics.summarization import SummarizationMetric

doc = """The full cost of damage in Newton Stewart, one of the areas worst affected, is still being assessed. Repair work is ongoing in Hawick and many roads in Peeblesshire remain badly affected by standing water. Trains on the west coast mainline face disruption due to damage at the Lamington Viaduct. Many businesses and householders were affected by flooding in Newton Stewart after the River Cree overflowed into the town. First Minister Nicola Sturgeon visited the area to inspect the damage. The waters breached a retaining wall, flooding many commercial properties on Victoria Street - the main shopping thoroughfare. Jeanette Tate, who owns the Cinnamon Cafe which was badly affected, said she could not fault the multi-agency response once the flood hit. However, she said more preventative work could have been carried out to ensure the retaining wall did not fail. "It is difficult but I do think there is so much publicity for Dumfries and the Nith - and I totally appreciate that - but it is almost like we're neglected or forgotten," she said. "That may not be true but it is perhaps my perspective over the last few days. "Why were you not ready to help us a bit more when the warning and the alarm alerts had gone out?" Meanwhile, a flood alert remains in place across the Borders because of the constant rain. Peebles was badly hit by problems, sparking calls to introduce more defences in the area. Scottish Borders Council has put a list on its website of the roads worst affected and drivers have been urged not to ignore closure signs. The Labour Party's deputy Scottish leader Alex Rowley was in Hawick on Monday to see the situation first hand. He said it was important to get the flood protection plan right but backed calls to speed up the process. "I was quite taken aback by the amount of damage that has been done," he said. "Obviously it is heart-breaking for people who have been forced out of their homes and the impact on businesses." He said it was important that "immediate steps" were taken to protect the areas most vulnerable and a clear timetable put in place for flood prevention plans. Have you been affected by flooding in Dumfries and Galloway or the Borders? Tell us about your experience of the situation and how it was handled. Email us on selkirk.news@bbc.co.uk or dumfries@bbc.co.uk."""

ref_sum = "Clean-up operations are continuing across the Scottish Borders and Dumfries and Galloway after flooding caused by Storm Frank."

sum_eval = SummarizationMetric(model="http://microservice_host:microservice_port/generate")

generation_config = {
        "max_new_tokens": 256,
        "do_sample": True}

sum_eval.summarize(doc, ref_sum, **generation_config)
print(sum_eval.average_score)

evaluation with LLMs (like openai-gpt4)

only evaluation

from evals.metrics.summarization import SummarizationMetric

sum_eval = SummarizationMetric(llm_judge="http://microservice_host:microservice_port/generate")

sum_eval.llm_scores(document, eval_summary)
print(sum_eval.average_score)

@kevinintel kevinintel linked an issue Jun 21, 2024 that may be closed by this pull request
@lkk12014402 lkk12014402 merged commit 3ec5441 into opea-project:main Jun 26, 2024
3 checks passed
lkk12014402 pushed a commit that referenced this pull request Sep 19, 2024
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Auto DocSum for accuracy
3 participants