Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history
Signed-off-by: arendu <adithyare@nvidia.com>
  • Loading branch information
arendu committed Oct 2, 2023
1 parent c6ed0b2 commit 1616f47
Showing 1 changed file with 6 additions and 14 deletions.
20 changes: 6 additions & 14 deletions scripts/metric_calculation/peft_metric_calc.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,26 +21,18 @@


"""
This script can be used to calcualte exact match and F1 scores for many different tasks, not just squad.
Example command for T5 Preds
```
python squad_metric_calc.py \
--ground-truth squad_test_gt.jsonl \
--preds squad_preds_t5.txt
```
This script can be used to calcualte exact match and F1 scores for many different tasks.
The file "squad_test_predictions.jsonl" is assumed to be generated by the
`examples/nlp/language_modeling/tuning/megatron_gpt_peft_eval.py` script
Example command for GPT Preds
```
python squad_metric_calc.py \
--ground-truth squad_test_gt.jsonl \
--preds squad_preds_gpt.txt \
--split-string "answer:"
python peft_metric_calc.py \
--pred_file squad_test_predictions.jsonl \
--label_field "original_answers" \
```
In this case, the prediction file will be split on "answer: " when looking for the LM's predicted answer.
"""

Expand Down

0 comments on commit 1616f47

Please sign in to comment.