You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There seems to currently be a mixture of OOP principles as well as functional principles throughout the codebase. Some of these seem like they can be combined.
For example, MMLUEvaluator implements all common functionality in a base abstract class which its variants then inherit.
However; MTBench seems to implement some base methods in the class, but also call out to generate_answers functionally, which seems counterintuitive to intended functionality.
There seems to currently be a mixture of OOP principles as well as functional principles throughout the codebase. Some of these seem like they can be combined.
For example, MMLUEvaluator implements all common functionality in a base abstract class which its variants then inherit.
However; MTBench seems to implement some base methods in the class, but also call out to
generate_answers
functionally, which seems counterintuitive to intended functionality.eval/src/instructlab/eval/mt_bench_answers.py
Line 97 in f4db222
We should combine this into the existing abstract class or make everuthing functional.
The text was updated successfully, but these errors were encountered: