-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Longbench: NV code to ipex-llm #11662
Merged
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
7f9e54f
add nv longbench
cyita 3abce54
LongBench: NV code to ipex-llm
bffdd3f
ammend
5ddeda9
add more models support
6e92d0d
ammend
2b3b82c
optimize LongBench's user experience
leonardozcm 6d805c6
ammend
leonardozcm 02aadc9
ammend
ATMxsp01 f120090
fix typo
ATMxsp01 ff0ec40
ammend
ATMxsp01 76ac859
remove cuda related information & add a readme
ATMxsp01 0aecd8d
add license to python scripts & polish the readme
1cef176
ammend
55e3a44
ammend
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,124 @@ | ||
# LongBench Benchmark Test | ||
|
||
LongBench is the first benchmark for bilingual, multitask, and comprehensive assessment of long context understanding capabilities of large language models. This benchmark implementation is adapted from [THUDM/LongBench](https://github.com/THUDM/LongBench) and [SnapKV/experiments/LongBench](https://github.com/FasterDecoding/SnapKV/tree/main/experiments/LongBench). | ||
|
||
|
||
## Environment Preparation | ||
|
||
Before running, make sure to have [ipex-llm](../../../../../README.md) installed. | ||
|
||
```bash | ||
pip install omegaconf | ||
pip install datasets | ||
pip install jieba | ||
pip install fuzzywuzzy | ||
pip install rouge | ||
``` | ||
|
||
### Load Data | ||
|
||
You can download and load the LongBench data through the Hugging Face datasets ([🤗 HF Repo](https://huggingface.co/datasets/THUDM/LongBench)): | ||
|
||
```python | ||
|
||
from datasets import load_dataset | ||
|
||
datasets = ["narrativeqa", "qasper", "multifieldqa_en", "multifieldqa_zh", "hotpotqa", "2wikimqa", "musique", \ | ||
"dureader", "gov_report", "qmsum", "multi_news", "vcsum", "trec", "triviaqa", "samsum", "lsht", \ | ||
"passage_count", "passage_retrieval_en", "passage_retrieval_zh", "lcc", "repobench-p"] | ||
|
||
for dataset in datasets: | ||
data = load_dataset('THUDM/LongBench', dataset, split='test') | ||
data = load_dataset('THUDM/LongBench', f"{dataset}_e", split='test') | ||
|
||
``` | ||
|
||
## config | ||
|
||
### `config.yaml` | ||
|
||
Config YAML file has following format | ||
|
||
```yaml | ||
# The name of the models you want to test | ||
model_name: | ||
# - "mistral-7B-instruct-v0.2" | ||
- "llama2-7b-chat-4k" | ||
# - "chatglm4-9b" | ||
# - "qwen2-7b-instruct" | ||
|
||
# whether test the full-kv | ||
full_kv: True | ||
# Whether apply model optimization | ||
optimize_model: True | ||
# dtype of the model | ||
dtype: 'fp16' | ||
# low bit of the model | ||
low_bit: 'sym_int4' | ||
# whether or not to use the 'e' version of the datasets | ||
e: False | ||
|
||
# the compress kv configs you want to test | ||
compress_kv: | ||
- "ablation_c512_w32_k7_maxpool" | ||
- "ablation_c1024_w32_k7_maxpool" | ||
|
||
# the datasets you want to test | ||
datasets: | ||
- "multi_news" | ||
- "qasper" | ||
- "hotpotqa" | ||
- "trec" | ||
- "passage_count" | ||
- "lcc" | ||
# - "multifieldqa_zh" | ||
# - "dureader" | ||
# - "vcsum" | ||
# - "lsht" | ||
# - "passage_retrieval_zh" | ||
|
||
``` | ||
|
||
### The `config` dir | ||
|
||
Some json files is saved in the `config` dir. It can be divided into three kinds: about models, about datasets, and about compress-kv. | ||
|
||
#### About Models | ||
|
||
- `model2path.json`: This file saves the path to the models. | ||
|
||
- `model2maxlen.json`: This file saves the max length of the prompts of each model. | ||
|
||
#### About datasets | ||
|
||
- `dataset2maxlen.json`: The max length of the outputs of the models of each dataset. | ||
|
||
- `dataset2prompt.json`: The format of prompts of each dataset. | ||
|
||
#### About compress-kv | ||
|
||
The rest JSON files are compress-kv test configurations. | ||
|
||
## Run | ||
|
||
There are two python files for users' call. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
||
1. Configure the `config.yaml` and run `pred.py` and you can obtain the output of the model under `pred/` folder corresponding to the model name. | ||
|
||
2. Run the evaluation code `eval.py`, you can get the evaluation results on all datasets in `result.json`. | ||
|
||
> [!Note] | ||
> | ||
> To test the models and get the score in a row, please run `test_and_eval.sh` | ||
|
||
## Citation | ||
|
||
```bibtex | ||
@article{bai2023longbench, | ||
title={LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding}, | ||
author={Bai, Yushi and Lv, Xin and Zhang, Jiajie and Lyu, Hongchang and Tang, Jiankai and Huang, Zhidian and Du, Zhengxiao and Liu, Xiao and Zeng, Aohan and Hou, Lei and Dong, Yuxiao and Tang, Jie and Li, Juanzi}, | ||
journal={arXiv preprint arXiv:2308.14508}, | ||
year={2023} | ||
} | ||
|
||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
model_name: | ||
# - "mistral-7B-instruct-v0.2" | ||
- "llama2-7b-chat-4k" | ||
# - "chatglm4-9b" | ||
# - "qwen2-7b-instruct" | ||
|
||
full_kv: True | ||
optimize_model: True | ||
dtype: 'fp16' | ||
low_bit: 'sym_int4' | ||
|
||
e: False | ||
|
||
compress_kv: | ||
- "ablation_c512_w32_k7_maxpool" | ||
- "ablation_c1024_w32_k7_maxpool" | ||
|
||
datasets: | ||
- "multi_news" | ||
- "qasper" | ||
- "hotpotqa" | ||
- "trec" | ||
- "passage_count" | ||
- "lcc" | ||
# - "multifieldqa_zh" | ||
# - "dureader" | ||
# - "vcsum" | ||
# - "lsht" | ||
# - "passage_retrieval_zh" |
7 changes: 7 additions & 0 deletions
7
python/llm/dev/benchmark/LongBench/config/ablation_c1024_w32_k7_maxpool.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
{ | ||
"window_sizes": 32, | ||
"default_max_capacity_prompts": 1024, | ||
"specific_max_capcity_prompts": {}, | ||
"kernel_sizes": 7, | ||
"pooling": "maxpool" | ||
} |
7 changes: 7 additions & 0 deletions
7
python/llm/dev/benchmark/LongBench/config/ablation_c2048_w32_k7_maxpool.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
{ | ||
"window_sizes": 32, | ||
"default_max_capacity_prompts": 2048, | ||
"specific_max_capcity_prompts": {}, | ||
"kernel_sizes": 7, | ||
"pooling": "maxpool" | ||
} |
7 changes: 7 additions & 0 deletions
7
python/llm/dev/benchmark/LongBench/config/ablation_c4096_w32_k7_maxpool.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
{ | ||
"window_sizes": 32, | ||
"default_max_capacity_prompts": 4096, | ||
"specific_max_capcity_prompts": {}, | ||
"kernel_sizes": 7, | ||
"pooling": "maxpool" | ||
} |
7 changes: 7 additions & 0 deletions
7
python/llm/dev/benchmark/LongBench/config/ablation_c512_w32_k7_maxpool.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
{ | ||
"window_sizes": 32, | ||
"default_max_capacity_prompts": 512, | ||
"specific_max_capcity_prompts": {}, | ||
"kernel_sizes": 7, | ||
"pooling": "maxpool" | ||
} |
23 changes: 23 additions & 0 deletions
23
python/llm/dev/benchmark/LongBench/config/dataset2maxlen.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
{ | ||
"narrativeqa": 128, | ||
"qasper": 128, | ||
"multifieldqa_en": 64, | ||
"multifieldqa_zh": 64, | ||
"hotpotqa": 32, | ||
"2wikimqa": 32, | ||
"musique": 32, | ||
"dureader": 128, | ||
"gov_report": 512, | ||
"qmsum": 512, | ||
"multi_news": 512, | ||
"vcsum": 512, | ||
"trec": 64, | ||
"triviaqa": 32, | ||
"samsum": 128, | ||
"lsht": 64, | ||
"passage_count": 32, | ||
"passage_retrieval_en": 32, | ||
"passage_retrieval_zh": 32, | ||
"lcc": 64, | ||
"repobench-p": 64 | ||
} |
23 changes: 23 additions & 0 deletions
23
python/llm/dev/benchmark/LongBench/config/dataset2prompt.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
{ | ||
"narrativeqa": "You are given a story, which can be either a novel or a movie script, and a question. Answer the question asconcisely as you can, using a single phrase if possible. Do not provide any explanation.\n\nStory: {context}\n\nNow, answer the question based on the story asconcisely as you can, using a single phrase if possible. Do not provide any explanation.\n\nQuestion: {input}\n\nAnswer:", | ||
"qasper": "You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write \"unanswerable\". If the question is a yes/no question, answer \"yes\", \"no\", or \"unanswerable\". Do not provide any explanation.\n\nArticle: {context}\n\n Answer the question based on the above article as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write \"unanswerable\". If the question is a yes/no question, answer \"yes\", \"no\", or \"unanswerable\". Do not provide any explanation.\n\nQuestion: {input}\n\nAnswer:", | ||
"multifieldqa_en": "Read the following text and answer briefly.\n\n{context}\n\nNow, answer the following question based on the above text, only give me the answer and do not output any other words.\n\nQuestion: {input}\nAnswer:", | ||
"multifieldqa_zh": "阅读以下文字并用中文简短回答:\n\n{context}\n\n现在请基于上面的文章回答下面的问题,只告诉我答案,不要输出任何其他字词。\n\n问题:{input}\n回答:", | ||
"hotpotqa": "Answer the question based on the given passages. Only give me the answer and do not output any other words.\n\nThe following are given passages.\n{context}\n\nAnswer the question based on the given passages. Only give me the answer and do not output any other words.\n\nQuestion: {input}\nAnswer:", | ||
"2wikimqa": "Answer the question based on the given passages. Only give me the answer and do not output any other words.\n\nThe following are given passages.\n{context}\n\nAnswer the question based on the given passages. Only give me the answer and do not output any other words.\n\nQuestion: {input}\nAnswer:", | ||
"musique": "Answer the question based on the given passages. Only give me the answer and do not output any other words.\n\nThe following are given passages.\n{context}\n\nAnswer the question based on the given passages. Only give me the answer and do not output any other words.\n\nQuestion: {input}\nAnswer:", | ||
"dureader": "请基于给定的文章回答下述问题。\n\n文章:{context}\n\n请基于上述文章回答下面的问题。\n\n问题:{input}\n回答:", | ||
"gov_report": "You are given a report by a government agency. Write a one-page summary of the report.\n\nReport:\n{context}\n\nNow, write a one-page summary of the report.\n\nSummary:", | ||
"qmsum": "You are given a meeting transcript and a query containing a question or instruction. Answer the query in one or more sentences.\n\nTranscript:\n{context}\n\nNow, answer the query based on the above meeting transcript in one or more sentences.\n\nQuery: {input}\nAnswer:", | ||
"multi_news": "You are given several news passages. Write a one-page summary of all news. \n\nNews:\n{context}\n\nNow, write a one-page summary of all the news.\n\nSummary:", | ||
"vcsum": "下面有一段会议记录,请你阅读后,写一段总结,总结会议的内容。\n会议记录:\n{context}\n\n会议总结:", | ||
"trec": "Please determine the type of the question below. Here are some examples of questions.\n\n{context}\n{input}", | ||
"triviaqa": "Answer the question based on the given passage. Only give me the answer and do not output any other words. The following are some examples.\n\n{context}\n\n{input}", | ||
"samsum": "Summarize the dialogue into a few short sentences. The following are some examples.\n\n{context}\n\n{input}", | ||
"lsht": "请判断给定新闻的类别,下面是一些例子。\n\n{context}\n{input}", | ||
"passage_count": "There are some paragraphs below sourced from Wikipedia. Some of them may be duplicates. Please carefully read these paragraphs and determine how many unique paragraphs there are after removing duplicates. In other words, how many non-repeating paragraphs are there in total?\n\n{context}\n\nPlease enter the final count of unique paragraphs after removing duplicates. The output format should only contain the number, such as 1, 2, 3, and so on.\n\nThe final answer is: ", | ||
"passage_retrieval_en": "Here are 30 paragraphs from Wikipedia, along with an abstract. Please determine which paragraph the abstract is from.\n\n{context}\n\nThe following is an abstract.\n\n{input}\n\nPlease enter the number of the paragraph that the abstract is from. The answer format must be like \"Paragraph 1\", \"Paragraph 2\", etc.\n\nThe answer is: ", | ||
"passage_retrieval_zh": "以下是若干段落文字,以及其中一个段落的摘要。请确定给定的摘要出自哪一段。\n\n{context}\n\n下面是一个摘要\n\n{input}\n\n请输入摘要所属段落的编号。答案格式必须是\"段落1\",\"段落2\"等格式\n\n答案是:", | ||
"lcc": "Please complete the code given below. \n{context}Next line of code:\n", | ||
"repobench-p": "Please complete the code given below. \n{context}{input}Next line of code:\n" | ||
} |
18 changes: 18 additions & 0 deletions
18
python/llm/dev/benchmark/LongBench/config/model2maxlen.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
{ | ||
cyita marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"llama2-7b-chat-4k": 4096, | ||
"longchat-v1.5-7b-32k": 4096, | ||
"xgen-7b-8k": 4096, | ||
"internlm-7b-8k": 4096, | ||
"chatglm2-6b": 4096, | ||
"chatglm2-6b-32k": 4096, | ||
"chatglm3-6b-32k": 4096, | ||
"chatglm4-9b": 4096, | ||
"vicuna-v1.5-7b-16k": 4096, | ||
"mistral-7B-instruct-v0.2": 4096, | ||
"mistral-7B-instruct-v0.1": 4096, | ||
"mixtral-8x7B-instruct-v0.1": 4096, | ||
"llama-2-7B-32k-instruct": 4096, | ||
"lwm-text-chat-1m": 4096, | ||
"lwm-text-1m": 4096, | ||
"qwen2-7b-instruct": 4096 | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
{ | ||
"llama2-7b-chat-4k": "meta-llama/Llama-2-7b-chat-hf", | ||
"longchat-v1.5-7b-32k": "lmsys/longchat-7b-v1.5-32k", | ||
"xgen-7b-8k": "Salesforce/xgen-7b-8k-inst", | ||
"internlm-7b-8k": "internlm/internlm-chat-7b-8k", | ||
"chatglm2-6b": "THUDM/chatglm2-6b", | ||
"chatglm2-6b-32k": "THUDM/chatglm2-6b-32k", | ||
"chatglm3-6b-32k": "THUDM/chatglm3-6b-32k", | ||
"chatglm4-9b": "THUDM/glm-4-9b-chat", | ||
"vicuna-v1.5-7b-16k": "lmsys/vicuna-7b-v1.5-16k", | ||
"mistral-7B-instruct-v0.2": "mistralai/Mistral-7B-Instruct-v0.2", | ||
"mistral-7B-instruct-v0.1": "mistralai/Mistral-7B-Instruct-v0.1", | ||
"mixtral-8x7B-instruct-v0.1": "mistralai/Mixtral-8x7B-Instruct-v0.1", | ||
"llama-2-7B-32k-instruct": "togethercomputer/Llama-2-7B-32K-Instruct", | ||
"lwm-text-chat-1m": "LargeWorldModel/LWM-Text-Chat-1M", | ||
"lwm-text-1m": "LargeWorldModel/LWM-Text-1M", | ||
"qwen2-7b-instruct": "Qwen/Qwen2-7B-Instruct" | ||
} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add Load Data section
You can download and load the LongBench data through the Hugging Face datasets (🤗 HF Repo):