Llama-2-70B FP8 quantization trust_remote_code=True not passed in llm_eval

[aalbersk]

System Info

Optimum-habana v1.13.2
HL-SMI: hl-1.17.1-fw-51.5.0
Driver: 1.17.1-78932ae

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. I created a Dockerfile to set up an environment.

FROM vault.habana.ai/gaudi-docker/1.17.1/ubuntu22.04/habanalabs/pytorch-installer-2.3.1:latest

ENV LANG=en_US.UTF-8
WORKDIR /root
RUN pip install --upgrade-strategy eager optimum[habana]
RUN pip install git+https://github.com/HabanaAI/DeepSpeed.git@1.17.0

RUN git clone https://github.com/huggingface/optimum-habana && cd optimum-habana && git checkout v1.13.2

WORKDIR /root/optimum-habana/examples/text-generation
RUN pip install -r requirements.txt
RUN pip install -r requirements_lm_eval.txt
CMD ["/bin/bash"]

2. I ran following command, created based on information in https://github.com/huggingface/optimum-habana/tree/main/examples/text-generation#running-with-fp8 .

QUANT_CONFIG=./quantization_config/maxabs_measure.json python ../gaudi_spawn.py \
--use_deepspeed --world_size 8 run_lm_eval.py \
--output_dir /data/Llama-2-70b-hf/ \
-o /data/Llama-2-70b-hf/acc_70b_bs1_measure.txt \
--model_name_or_path meta-llama/Llama-2-70b-hf \
--attn_softmax_bf16 \
--use_hpu_graphs \
--trim_logits \
--use_kv_cache \
--bucket_size=128 \
--bucket_internal \
--use_flash_attention \
--flash_attention_recompute \
--bf16 \
--trust_remote_code \
--batch_size 1

I received information that even with a parameter --trust_remote_code llm_eval didn't receive this parameter and failed on datasets.load_dataset() due o lack of trust_remote_code=True.

Error:

[rank2]: Traceback (most recent call last):
[rank2]:   File "/root/optimum-habana/examples/text-generation/run_lm_eval.py", line 229, in <module>
[rank2]:     main()
[rank2]:   File "/root/optimum-habana/examples/text-generation/run_lm_eval.py", line 197, in main
[rank2]:     lm_tasks = lm_eval.tasks.get_task_dict(args.tasks)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 415, in get_task_dict
[rank2]:     task_name_dict = {
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 416, in <dictcomp>
[rank2]:     task_name: get_task(task_name)()
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/base.py", line 513, in __init__
[rank2]:     self.download(data_dir, cache_dir, download_mode)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/base.py", line 542, in download
[rank2]:     self.dataset = datasets.load_dataset(
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2606, in load_dataset
[rank2]:     builder_instance = load_dataset_builder(
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2277, in load_dataset_builder
[rank2]:     dataset_module = dataset_module_factory(
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1923, in dataset_module_factory
[rank2]:     raise e1 from None
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1896, in dataset_module_factory
[rank2]:     ).get_module()
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1545, in get_module
[rank2]:     raise ValueError(
[rank2]: ValueError: Loading winogrande requires you to execute the dataset script in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.

Expected behavior

I expected that the dataset would be loaded without any issues and a model will start static quantization.

[regisss]

It's an issue in the lm_eval library, not in Optimum Habana. It seems you can fix it that way: EleutherAI/lm-evaluation-harness#2222