Skip to content

Commit

Permalink
Update docstring for EleutherEvalRecipe
Browse files Browse the repository at this point in the history
  • Loading branch information
joecummings committed Mar 22, 2024
1 parent 74d8bf8 commit 2d30e98
Showing 1 changed file with 17 additions and 1 deletion.
18 changes: 17 additions & 1 deletion recipes/eleuther_eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,23 @@ def _model_generate(self, *args, **kwargs):


class EleutherEvalRecipe(EvalRecipeInterface):
"""This recipe runs evaluation on a trained model using EleutherAI's eval harness."""
"""This recipe runs evaluation on a trained model using EleutherAI's eval harness.
This recipe supports:
- Single GPU evaluation
- Loading model in fp32 or bf16
- Any task from the EleutherAI eval harness that is not free generation
Assumptions:
- Evaluation is launched with the Tune CLI (recommended)
- User has the EleutherAI eval harness installed, see https://github.com/EleutherAI/lm-evaluation-harness
The following configs can be used to run this recipe:
- llama2_eleuther_eval.yaml
Args:
cfg (DictConfig): OmegaConf object parsed from YAML file
"""

def __init__(self, cfg: DictConfig) -> None:
self._cfg = cfg
Expand Down

0 comments on commit 2d30e98

Please sign in to comment.