Offload Folder Set Up

[kbzh2558]

Hi,

When running the model agent, I encountered the following error msg: ValueError: The current device_map had weights offloaded to the disk. Please provide an offload_folder for them. Alternatively, make sure you have safetensors installed if the model you are using offers the weights in this format.

I tried to look it up but have not found a way to set up this folder. Thank you! The detailed code of the agent set up is as followed:

rom modelscope_agent.agents.role_play import RolePlay

role_template = """
你是一位精通金融信息分析的ai助手。
你需要遵守的规则是:
1. 必须使用中文结合查询的背景信息结合你所拥有的知识回答用户提出的问题。
2. 结构化答案生成，必要时通过空行提升阅读体验。
3. 不采用背景信息中的错误信息。
4. 要考虑答案和问题的相关性，不做对问题没有帮助的回答。
5. 详尽回答问题，重点突出，不过多花哨词藻。
6. 不说模糊的推测。
7. 尽量多的使用数值类信息。
"""
llm_config = {'model': 'ZhipuAI/chatglm3-6b', 'model_server': 'modelscope'}

input tool name

function_list = ['get-index']

storage_path

offload_folder = 'D:\PPT&WORD\folder\model_training'

bot = RolePlay(
function_list=function_list, llm=llm_config, instruction=role_template, storage_path=offload_folder)

[mushenL]

Hello, it seems that you have chosen "modelscope" as your model server. However, please note that this model server type has not been maintained and adapted recently. Additionally, different models have different input and output formats, which makes maintenance difficult. We suggest that you use the VLLM deployment service to unify the format. You can use the following code:

import os
from modelscope.hub.snapshot_download import snapshot_download

os.environ['MODELSCOPE_CACHE'] = "/mnt/workspace/yangkun/yk_cache"

model_dir = snapshot_download('ZhipuAI/chatglm3-6b')
print(model_dir)

The above code allows downloading the model from ModelScope.
bash scripts/run_assistant_server.sh --served-model-name chatglm3-6b --model your model_dir
Using the above commands, you can deploy a VLLM-based agent server locally. You can use the following code to verify if your service has been successfully deployed:

curl -X POST 'http://localhost:31512/v1/chat/completions'
-H 'Content-Type: application/json'
-d '{
"tools": [{
"type": "function",
"function": {
"name": "amap_weather",
"description": "amap weather tool",
"parameters": [{
"name": "location",
"type": "string",
"description": "城市/区具体名称，如北京市海淀区请描述为海淀区",
"required": true
}]
}
}],
"tool_choice": "auto",
"model": "chatglm3-6b",
"messages": [
{"content": "海淀区天气", "role": "user"}
]
}'

For detailed documentation, please refer to this link