[BUGFIX] Adding sequence truncation to max_seq_length in eval recipe

[SalmanMohammadi]

Context

What is the purpose of this PR? Is it to

• add a new feature
• fix a bug
• update tests and/or documentation
• other (please add here)

closes #1644

Evidence of running a recipe with max_seq_length=320 and model.max_seq_len=320, which shows that recipe correctly truncated longer sequences.

(tune) salman@combuter:~/torchtune$ tune run eleuther_eval --config target/eleuther_evaluation.yaml 
2024-10-08:23:34:32,751 INFO     [_logging.py:101] Running EleutherEvalRecipe with resolved config:

batch_size: 4
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: ./target/1b_normal
  checkpoint_files:
  - pytorch_model.bin
  model_type: LLAMA2
  output_dir: ./target/tmp
device: cuda
dtype: bf16
enable_kv_cache: true
limit: 20
max_seq_length: 320
model:
  _component_: torchtune.models.llama2.llama2
  embed_dim: 2048
  max_seq_len: 320
  norm_eps: 1.0e-05
  num_heads: 32
  num_kv_heads: 4
  num_layers: 22
  vocab_size: 32000
quantizer: null
seed: 1234
tasks:
- truthfulqa_gen
tokenizer:
  _component_: torchtune.models.llama2.llama2_tokenizer
  path: ./target/1b_normal/tokenizer.model

2024-10-08:23:34:34,267 INFO     [eleuther_eval.py:512] Model is initialized with precision torch.bfloat16.
2024-10-08:23:34:34,291 INFO     [huggingface.py:132] Using device 'cuda:0'
/home/salman/.pyenv/versions/3.11.9/envs/tune/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
2024-10-08:23:34:34,825 INFO     [huggingface.py:368] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
2024-10-08:23:34:36,264 INFO     [__init__.py:491] `group` and `group_alias` keys in TaskConfigs are deprecated and will be removed in v0.4.5 of lm_eval. The new `tag` field will be used to allow for a shortcut to a group of tasks one does not wish to aggregate metrics across. `group`s which aggregate across subtasks must be only defined in a separate group config file, which will be the official way to create groups that support cross-task aggregation as in `mmlu`. Please see the v0.4.4 patch notes and our documentation: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#advanced-group-configs for more information.
2024-10-08:23:34:46,406 INFO     [eleuther_eval.py:556] Running evaluation on the following tasks: ['truthfulqa_gen']
2024-10-08:23:34:46,408 INFO     [task.py:428] Building contexts for truthfulqa_gen on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 1516.74it/s]
2024-10-08:23:34:46,426 INFO     [evaluator.py:485] Running generate_until requests
Running generate_until requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [01:15<00:00,  3.78s/it]
2024-10-08:23:36:02,095 INFO     [rouge_scorer.py:83] Using default tokenizer.
2024-10-08:23:36:21,072 INFO     [eleuther_eval.py:565] Eval completed in 94.67 seconds.
2024-10-08:23:36:21,072 INFO     [eleuther_eval.py:566] Max memory allocated: 2.77 GB
2024-10-08:23:36:21,180 INFO     [eleuther_eval.py:570] 

|    Tasks     |Version|Filter|n-shot|  Metric   |   | Value |   |Stderr|
|--------------|------:|------|-----:|-----------|---|------:|---|-----:|
|truthfulqa_gen|      3|none  |     0|bleu_acc   |↑  | 0.2500|±  |0.0993|
|              |       |none  |     0|bleu_diff  |↑  |-5.9765|±  |2.8679|
|              |       |none  |     0|bleu_max   |↑  |17.7870|±  |3.8351|
|              |       |none  |     0|rouge1_acc |↑  | 0.3500|±  |0.1094|
|              |       |none  |     0|rouge1_diff|↑  |-4.1224|±  |3.3162|
|              |       |none  |     0|rouge1_max |↑  |41.5919|±  |5.6237|
|              |       |none  |     0|rouge2_acc |↑  | 0.2000|±  |0.0918|
|              |       |none  |     0|rouge2_diff|↑  |-9.1829|±  |3.2808|
|              |       |none  |     0|rouge2_max |↑  |23.0919|±  |5.6656|
|              |       |none  |     0|rougeL_acc |↑  | 0.3000|±  |0.1051|
|              |       |none  |     0|rougeL_diff|↑  |-5.4918|±  |3.3711|
|              |       |none  |     0|rougeL_max |↑  |38.8588|±  |5.8452|

Test plan

Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.

• run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
• add unit tests for any new functionality
• update docstrings for any new or updated methods or classes
• run unit tests via pytest tests
• run recipe tests via pytest tests -m integration_test
• manually run any new or modified recipes with sufficient proof of correctness
• include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

UX

If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Here is a docstring example
and a tutorial example

• I did not change any public API
• I have added an example to docs or docstrings

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1773

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 6f95550 with merge base 27b0fcc ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.