[Question]: 本地部署Qwen2后的文件上传问题

[Patrick24080735]

Model Series

Qwen2

What are the models used?

Qwen2-7B-Instruct

What is the scenario where the problem happened?

Files like json can not be uploaded.

Is this badcase known and can it be solved using avaiable techniques?

• I have followed the GitHub README.
• I have checked the Qwen documentation and cannot find a solution there.
• I have checked the documentation of the related framework and cannot find useful information.
• I have searched the issues and there is not a similar one.

Information about environment

Ubuntu20.04 A6000*4 python3.11

Description

我想批量处理数据，但是没有办法在本地部署的大模型上传文件，client和webui都没有上传选项，如果本地部署不能上传文件，请告知我；或者有其他方法上传，请告知我，谢谢！

[jklj077]

what do you mean by uploading files? do you mean RAG? you need a rag file for that.

[Patrick24080735]

No, I wanna do some knowledge extractions from paragraphs, but my input data is huge, so I wonder if there are some ways to tackle the problem, uploading data files I guess? If you know something, please tell me, thank you.

[jklj077]

GUIs are designed for interactivity for LLMs.

If you have large quantities of examples and you want to run them without manually starting each one, you should use the programatic interface. A simple for-loop inference is mostly enough with a vLLM backend.

[thiner]

本地用Ollama部署的吗？你需要再部署一个客户端，比如Open Webui, LobeChat之类，连接本地部署的模型API。这样就可以在网页应用中调用本地模型处理文件了。

[Patrick24080735]

我现在的打算是写一个脚本，用vLLM推理，但是推理完之后仅仅把数据的内容抽取出来了，需要自定义一个函数从抽取后的结果找到三元组，但是目前脚本的效果不太好，我在改进一下，谢谢。