Comprehensive Benchmark Example for a Coding Bot #49
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer's Guide by SourceryThis PR introduces a comprehensive benchmark suite for evaluating Large Language Models (LLMs), with initial support for the MMLU (Massive Multitask Language Understanding) benchmark. The implementation uses async/await patterns for efficient execution and includes features for downloading datasets, running evaluations, and generating detailed reports. Class diagram for the new benchmark suiteclassDiagram
class BenchmarkType {
<<enumeration>>
MMLU
GLUE
SUPERGLUE
HUMANEVAL
GSM8K
BIGBENCH
HELM
}
class BenchmarkResult {
float score
Dict metadata
List strengths
List weaknesses
}
class MMluQuestion {
string question
List choices
string correct_answer
string subject
}
class BenchmarkTask {
<<abstract>>
+evaluate(response: str) BenchmarkResult
+generate_prompt() str
}
class MMluTask {
-Path dataset_path
-List questions
-bool download_needed
+initialize() void
+generate_prompt() str
+evaluate(response: str) BenchmarkResult
+download_dataset(target_dir: str) void
+_load_questions() List
}
class LLMBenchmarkSuite {
-string model_name
-float temperature
-float top_p
-int top_k
-int questions_per_task
-Dict tasks
-Dict results
-AsyncClient client
+_get_llm_response(prompt: str) str
+add_benchmark(benchmark_type: BenchmarkType, tasks: List) void
+run_benchmarks(selected_types: Optional[List]) Dict
+generate_report() str
}
BenchmarkTask <|-- MMluTask
LLMBenchmarkSuite --> BenchmarkType
LLMBenchmarkSuite --> BenchmarkTask
LLMBenchmarkSuite --> BenchmarkResult
MMluTask --> MMluQuestion
MMluTask --> BenchmarkResult
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey @leonvanbokhorst - I've reviewed your changes - here's some feedback:
Overall Comments:
- Consider implementing pagination or streaming for dataset loading to reduce memory usage, especially for large benchmark datasets.
- Add proper timeout handling and retry logic in _get_llm_response() to handle network issues and API failures gracefully.
Here's what I looked at during the review
- 🟡 General issues: 1 issue found
- 🟢 Security: all looks good
- 🟢 Testing: all looks good
- 🟢 Complexity: all looks good
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| results.append(result) | ||
|
|
||
| self.results[benchmark_type] = results | ||
| scores[benchmark_type.name] = sum(r.score for r in results) / len(results) |
There was a problem hiding this comment.
issue: Missing edge case handling for empty task lists could cause division by zero
Add a check for empty results list before calculating the average score
Comment on lines
+117
to
+125
| for _, row in df.iterrows(): | ||
| questions.append( | ||
| MMluQuestion( | ||
| question=row[0], | ||
| choices=[row[1], row[2], row[3], row[4]], | ||
| correct_answer=row[5], | ||
| subject=subject, | ||
| ) | ||
| ) |
There was a problem hiding this comment.
issue (code-quality): Replace a for append loop with list extend (for-append-to-extend)
Comment on lines
+158
to
+164
| # Find first valid answer in response | ||
| answer = None | ||
| for char in response: | ||
| if char in valid_answers: | ||
| answer = char | ||
| break | ||
|
|
There was a problem hiding this comment.
suggestion (code-quality): Use the built-in function next instead of a for-loop (use-next)
Suggested change
| # Find first valid answer in response | |
| answer = None | |
| for char in response: | |
| if char in valid_answers: | |
| answer = char | |
| break | |
| answer = next((char for char in response if char in valid_answers), None) |
| total_score = 0 | ||
|
|
||
| for i, result in enumerate(results, 1): | ||
| report.append(f"\nTask {i}:") |
There was a problem hiding this comment.
issue (code-quality): We've found these issues:
- Merge consecutive list appends into a single extend (
merge-list-appends-into-extend) - Replace a for append loop with list extend [×3] (
for-append-to-extend)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #48
Summary by Sourcery
New Features: