needlebench llama-2-7B-hf offline: how can i evaluate llama-2-7B-hf in needlebench?

[ainilian]

(vllm) root@89c023b6aa76:/home/zzzxl/opencompass-main# CUDA_VISIBLE_DEVICES=4 python run.py --dataset needlebench_single_4k --models hf_llama2_7b --model-kwargs trust_remote_code=True --summarizer needlebench/needlebench_4k_summarizer  --hf-num-gpus 1
09/02 12:44:22 - OpenCompass - WARNING - Found ambiguous patterns, using the first matched config.
+-----------------------+---------------------------------------------------------------------------------------------------------------------+
| Ambiguous patterns    | Matched files                                                                                                       |
|-----------------------+---------------------------------------------------------------------------------------------------------------------|
| needlebench_single_4k | configs/datasets/needlebench/needlebench_4k/needlebench_single_4k.py                                                |
|                       | /home/zzzxl/opencompass-main/opencompass/configs/./datasets/needlebench/needlebench_4k/needlebench_single_4k.py |
+-----------------------+---------------------------------------------------------------------------------------------------------------------+
09/02 12:44:22 - OpenCompass - INFO - Loading needlebench_single_4k: configs/datasets/needlebench/needlebench_4k/needlebench_single_4k.py
09/02 12:44:22 - OpenCompass - WARNING - Found ambiguous patterns, using the first matched config.
+----------------------+----------------------------------------------------------------------------------------+
| Ambiguous patterns   | Matched files                                                                          |
|----------------------+----------------------------------------------------------------------------------------|
| hf_llama2_7b         | configs/models/hf_llama/hf_llama2_7b.py                                                |
|                      | /home/zzzxl/opencompass-main/opencompass/configs/./models/hf_llama/hf_llama2_7b.py |
+----------------------+----------------------------------------------------------------------------------------+
09/02 12:44:22 - OpenCompass - INFO - Loading hf_llama2_7b: configs/models/hf_llama/hf_llama2_7b.py
09/02 12:44:22 - OpenCompass - WARNING - Found ambiguous patterns, using the first matched config.
+----------------------+-----------------------------------------------------------------------------------+
| Ambiguous patterns   | Matched files                                                                     |
|----------------------+-----------------------------------------------------------------------------------|
| needlebench          | configs/summarizers/needlebench.py                                                |
|                      | /home/zzzxl/opencompass-main/opencompass/configs/./summarizers/needlebench.py |
+----------------------+-----------------------------------------------------------------------------------+
09/02 12:44:22 - OpenCompass - INFO - Loading needlebench: configs/summarizers/needlebench.py
09/02 12:44:22 - OpenCompass - INFO - Current exp folder: outputs/default/20240902_124422
09/02 12:44:22 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zzzxl/opencompass-main/run.py", line 4, in <module>
    main()
  File "/home/zzzxl/opencompass-main/opencompass/cli/main.py", line 299, in main
    tasks = partitioner(cfg)
  File "/home/zzzxl/opencompass-main/opencompass/partitioners/base.py", line 83, in __call__
    tasks = self.partition(**model_and_dataset_args,
  File "/home/zzzxl/opencompass-main/opencompass/partitioners/num_worker.py", line 69, in partition
    dataset_size = self.get_size(dataset)
  File "/home/zzzxl/opencompass-main/opencompass/partitioners/num_worker.py", line 141, in get_size
    dataset = build_dataset_from_cfg(dataset)
  File "/home/zzzxl/opencompass-main/opencompass/utils/build.py", line 13, in build_dataset_from_cfg
    return LOAD_DATASET.build(dataset_cfg)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/home/zzzxl/opencompass-main/opencompass/datasets/base.py", line 12, in __init__
    self.dataset = self.load(**kwargs)
  File "/home/zzzxl/opencompass-main/opencompass/datasets/needlebench/origin.py", line 52, in load
    tokenizer = tiktoken.encoding_for_model(tokenizer_model)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/tiktoken/model.py", line 103, in encoding_for_model
    return get_encoding(encoding_name_for_model(model_name))
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/tiktoken/registry.py", line 73, in get_encoding
    enc = Encoding(**constructor())
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/tiktoken_ext/openai_public.py", line 72, in cl100k_base
    mergeable_ranks = load_tiktoken_bpe(
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/tiktoken/load.py", line 147, in load_tiktoken_bpe
    contents = read_file_cached(tiktoken_bpe_file, expected_hash)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/tiktoken/load.py", line 64, in read_file_cached
    contents = read_file(blobpath)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/tiktoken/load.py", line 25, in read_file
    resp = requests.get(blobpath)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))

I have downloaded the model hf_llama2_7b and the needlebench dataset in .data/needlebench.

configs/models/hf_llama/hf_llama2_7b.py

from opencompass.models import HuggingFaceBaseModel

models = [
    dict(
        type=HuggingFaceBaseModel,
        abbr='llama-2-7b-hf',
        path='/base_models/llama-2-7b-hf',
        max_seq_len=32768,
        max_out_len=2000,
        batch_size=8,
        stop_words='</s>',
        run_cfg=dict(num_gpus=1),
    )
]

configs/datasets/needlebench/needlebench_4k/needlebench_single_4k.py

from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import GenInferencer
from opencompass.datasets.needlebench.origin import NeedleBenchOriginDataset
from opencompass.datasets.needlebench.origin import NeedleBenchOriginEvaluator
from opencompass.datasets.needlebench.origin import needlebench_postprocess
from opencompass.datasets.needlebench.origin import needlebench_dataset_postprocess
import math


def logistic(x, L=100, x0=50, k=0.1):
    return round(L / (1 + math.exp(-k * (x - x0))), 3)


def generate_linear_space(start, end, num):
    if num == 1:
        return [start]
    elif num < 1:
        raise ValueError('num must be at least 1.')
    step = (end - start) / (num - 1)
    return [start + step * i for i in range(num)]


def generate_depth_percents(intervals, interval_type):
    if interval_type == 'linear':
        return generate_linear_space(0, 100, intervals)
    elif interval_type == 'sigmoid':
        linear_space = generate_linear_space(0, 100, intervals)
        return [logistic(x) for x in linear_space]
    else:
        raise ValueError('Unsupported interval type')


needlebench_reader_cfg = dict(input_columns=['prompt'], output_column='answer')

needlebench_infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template=dict(
            round=[
                dict(role='HUMAN', prompt='{prompt}'),
                dict(role='BOT', prompt='{answer}\n'),
            ]
        )
        ),
    retriever=dict(type=ZeroRetriever),
    inferencer=dict(type=GenInferencer))

needlebench_eval_cfg = dict(
    evaluator=dict(type=NeedleBenchOriginEvaluator),
    pred_postprocessor=dict(type=needlebench_postprocess),
    dataset_postprocessor=dict(type=needlebench_dataset_postprocess),
    pred_role='BOT')

context_lengths = list(range(1000, 5000, 1000))
document_depth_percent_intervals = 20
document_depth_percent_interval_type = 'linear'

# base_path = './data/needlebench'
base_path = '/home/zzzxl/opencompass-main/data/needlebench'
file_list = ['PaulGrahamEssays.jsonl']
needlebench_en_datasets = []
needle_file_name = 'needles.jsonl'

for original_context_length in context_lengths:
    for depth_percent in generate_depth_percents(
            document_depth_percent_intervals,
            document_depth_percent_interval_type):
        dataset_dict = {
            'abbr': f'Length{original_context_length}'
                    f'Depth{int(depth_percent)}_origin_en_4k',
            'type': NeedleBenchOriginDataset,
            'path': base_path,
            'length': original_context_length,
            'depth': int(depth_percent),
            'tokenizer_model': 'gpt-4',
            'file_list': file_list,
            'num_repeats_per_file': 10,
            'length_buffer': 600,
            'guide': True,
            'language': 'English',
            'needle_file_name': needle_file_name,
            'reader_cfg': needlebench_reader_cfg,
            'infer_cfg': needlebench_infer_cfg,
            'eval_cfg': needlebench_eval_cfg
        }
        needlebench_en_datasets.append(dataset_dict)

file_list = ['zh_finance.jsonl']
needlebench_zh_datasets = []
needle_file_name = 'needles.jsonl'

for original_context_length in context_lengths:
    for depth_percent in generate_depth_percents(
            document_depth_percent_intervals,
            document_depth_percent_interval_type):
        dataset_dict = {
            'abbr': f'Length{original_context_length}'
                    f'Depth{int(depth_percent)}_origin_zh_4k',
            'type': NeedleBenchOriginDataset,
            'path': base_path,
            'length': original_context_length,
            'depth': int(depth_percent),
            'tokenizer_model': 'gpt-4',
            'file_list': file_list,
            'num_repeats_per_file': 10,
            'length_buffer': 200,
            'guide': True,
            'language': 'Chinese',
            'needle_file_name': needle_file_name,
            'reader_cfg': needlebench_reader_cfg,
            'infer_cfg': needlebench_infer_cfg,
            'eval_cfg': needlebench_eval_cfg
        }
        needlebench_zh_datasets.append(dataset_dict)

[tonysy]

It seems that this issue is related to network. tiktoken needs network to download tokenizer.
You can try https://stackoverflow.com/questions/76106366/how-to-use-tiktoken-in-offline-mode-computer this

[ainilian]

It's work! Thank you very much.