+ 
-
+
@@ -176,14 +176,14 @@ releases! 🌟
```
> - To download a RAGFlow slim Docker image of a specific version, update the `RAGFLOW_IMAGE` variable in *
- *docker/.env** to your desired version. For example, `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.0-slim`. After
+ *docker/.env** to your desired version. For example, `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1-slim`. After
making this change, rerun the command above to initiate the download.
> - To download the dev version of RAGFlow Docker image *including* embedding models and Python libraries, update the
`RAGFLOW_IMAGE` variable in **docker/.env** to `RAGFLOW_IMAGE=infiniflow/ragflow:dev`. After making this change,
rerun the command above to initiate the download.
> - To download a specific version of RAGFlow Docker image *including* embedding models and Python libraries, update
the `RAGFLOW_IMAGE` variable in **docker/.env** to your desired version. For example,
- `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.0`. After making this change, rerun the command above to initiate the
+ `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1`. After making this change, rerun the command above to initiate the
download.
> **NOTE:** A RAGFlow Docker image that includes embedding models and Python libraries is approximately 9GB in size
@@ -333,8 +333,7 @@ docker build -f Dockerfile -t infiniflow/ragflow:dev .
cd web
npm install --force
```
-7. Configure frontend to update `proxy.target` in **.umirc.ts** to `http://127.0.0.1:9380`:
-8. Launch frontend service:
+7. Launch frontend service:
```bash
npm run dev
```
diff --git a/README_id.md b/README_id.md
index 0b739d42548..3fa6262c068 100644
--- a/README_id.md
+++ b/README_id.md
@@ -20,7 +20,7 @@
-
+
@@ -169,14 +169,14 @@ Coba demo kami di [https://demo.ragflow.io](https://demo.ragflow.io).
```
> - Untuk mengunduh versi tertentu dari image Docker RAGFlow slim, perbarui variabel `RAGFlow_IMAGE` di *
- *docker/.env** sesuai dengan versi yang diinginkan. Misalnya, `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.0-slim`.
+ *docker/.env** sesuai dengan versi yang diinginkan. Misalnya, `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1-slim`.
Setelah mengubah ini, jalankan ulang perintah di atas untuk memulai unduhan.
> - Untuk mengunduh versi dev dari image Docker RAGFlow *termasuk* model embedding dan library Python, perbarui
variabel `RAGFlow_IMAGE` di **docker/.env** menjadi `RAGFLOW_IMAGE=infiniflow/ragflow:dev`. Setelah mengubah ini,
jalankan ulang perintah di atas untuk memulai unduhan.
> - Untuk mengunduh versi tertentu dari image Docker RAGFlow *termasuk* model embedding dan library Python, perbarui
variabel `RAGFlow_IMAGE` di **docker/.env** sesuai dengan versi yang diinginkan. Misalnya,
- `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.0`. Setelah mengubah ini, jalankan ulang perintah di atas untuk memulai unduhan.
+ `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1`. Setelah mengubah ini, jalankan ulang perintah di atas untuk memulai unduhan.
> **CATATAN:** Image Docker RAGFlow yang mencakup model embedding dan library Python berukuran sekitar 9GB
dan mungkin memerlukan waktu lebih lama untuk dimuat.
@@ -307,9 +307,8 @@ docker build -f Dockerfile -t infiniflow/ragflow:dev .
```bash
cd web
npm install --force
- ```
-7. Konfigurasikan frontend untuk memperbarui `proxy.target` di **.umirc.ts** menjadi `http://127.0.0.1:9380`:
-8. Jalankan aplikasi frontend:
+ ```
+7. Jalankan aplikasi frontend:
```bash
npm run dev
```
diff --git a/README_ja.md b/README_ja.md
index 28f8ccf2c57..e1854d1547b 100644
--- a/README_ja.md
+++ b/README_ja.md
@@ -20,7 +20,7 @@
-
+
@@ -148,9 +148,9 @@
$ docker compose -f docker-compose.yml up -d
```
- > - 特定のバージョンのRAGFlow slim Dockerイメージをダウンロードするには、**docker/.env**内の`RAGFlow_IMAGE`変数を希望のバージョンに更新します。例えば、`RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.0`とします。この変更を行った後、上記のコマンドを再実行してダウンロードを開始してください。
+ > - 特定のバージョンのRAGFlow slim Dockerイメージをダウンロードするには、**docker/.env**内の`RAGFlow_IMAGE`変数を希望のバージョンに更新します。例えば、`RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1`とします。この変更を行った後、上記のコマンドを再実行してダウンロードを開始してください。
> - RAGFlowの埋め込みモデルとPythonライブラリを含む開発版Dockerイメージをダウンロードするには、**docker/.env**内の`RAGFlow_IMAGE`変数を`RAGFLOW_IMAGE=infiniflow/ragflow:dev`に更新します。この変更を行った後、上記のコマンドを再実行してダウンロードを開始してください。
- > - 特定のバージョンのRAGFlow Dockerイメージ（埋め込みモデルとPythonライブラリを含む）をダウンロードするには、**docker/.env**内の`RAGFlow_IMAGE`変数を希望のバージョンに更新します。例えば、`RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.0`とします。この変更を行った後、上記のコマンドを再実行してダウンロードを開始してください。
+ > - 特定のバージョンのRAGFlow Dockerイメージ（埋め込みモデルとPythonライブラリを含む）をダウンロードするには、**docker/.env**内の`RAGFlow_IMAGE`変数を希望のバージョンに更新します。例えば、`RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1`とします。この変更を行った後、上記のコマンドを再実行してダウンロードを開始してください。
> **NOTE:** 埋め込みモデルとPythonライブラリを含むRAGFlow Dockerイメージのサイズは約9GBであり、読み込みにかなりの時間がかかる場合があります。
@@ -289,8 +289,7 @@ docker build -f Dockerfile -t infiniflow/ragflow:dev .
cd web
npm install --force
```
-7. フロントエンドを設定し、**.umirc.ts** の `proxy.target` を `http://127.0.0.1:9380` に更新します:
-8. フロントエンドサービスを起動する:
+7. フロントエンドサービスを起動する:
```bash
npm run dev
```
diff --git a/README_ko.md b/README_ko.md
index f558f4c75d8..0ad751e21e3 100644
--- a/README_ko.md
+++ b/README_ko.md
@@ -20,7 +20,7 @@
-
+
@@ -152,9 +152,9 @@
$ docker compose -f docker-compose.yml up -d
```
- > - 특정 버전의 RAGFlow slim Docker 이미지를 다운로드하려면, **docker/.env**에서 `RAGFlow_IMAGE` 변수를 원하는 버전으로 업데이트하세요. 예를 들어, `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.0-slim`으로 설정합니다. 이 변경을 완료한 후, 위의 명령을 다시 실행하여 다운로드를 시작하세요.
+ > - 특정 버전의 RAGFlow slim Docker 이미지를 다운로드하려면, **docker/.env**에서 `RAGFlow_IMAGE` 변수를 원하는 버전으로 업데이트하세요. 예를 들어, `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1-slim`으로 설정합니다. 이 변경을 완료한 후, 위의 명령을 다시 실행하여 다운로드를 시작하세요.
> - RAGFlow의 임베딩 모델과 Python 라이브러리를 포함한 개발 버전 Docker 이미지를 다운로드하려면, **docker/.env**에서 `RAGFlow_IMAGE` 변수를 `RAGFLOW_IMAGE=infiniflow/ragflow:dev`로 업데이트하세요. 이 변경을 완료한 후, 위의 명령을 다시 실행하여 다운로드를 시작하세요.
- > - 특정 버전의 RAGFlow Docker 이미지를 임베딩 모델과 Python 라이브러리를 포함하여 다운로드하려면, **docker/.env**에서 `RAGFlow_IMAGE` 변수를 원하는 버전으로 업데이트하세요. 예를 들어, `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.0` 로 설정합니다. 이 변경을 완료한 후, 위의 명령을 다시 실행하여 다운로드를 시작하세요.
+ > - 특정 버전의 RAGFlow Docker 이미지를 임베딩 모델과 Python 라이브러리를 포함하여 다운로드하려면, **docker/.env**에서 `RAGFlow_IMAGE` 변수를 원하는 버전으로 업데이트하세요. 예를 들어, `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1` 로 설정합니다. 이 변경을 완료한 후, 위의 명령을 다시 실행하여 다운로드를 시작하세요.
> **NOTE:** 임베딩 모델과 Python 라이브러리를 포함한 RAGFlow Docker 이미지의 크기는 약 9GB이며, 로드하는 데 상당히 오랜 시간이 걸릴 수 있습니다.
@@ -291,8 +291,7 @@ docker build -f Dockerfile -t infiniflow/ragflow:dev .
cd web
npm install --force
```
-7. **.umirc.ts** 에서 `proxy.target` 을 `http://127.0.0.1:9380` 으로 업데이트합니다:
-8. 프론트엔드 서비스를 시작합니다:
+7. 프론트엔드 서비스를 시작합니다:
```bash
npm run dev
```
diff --git a/README_zh.md b/README_zh.md
index e06e7feae6f..ee22fa597d6 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -20,7 +20,7 @@
-
+
@@ -149,9 +149,9 @@
$ docker compose -f docker-compose.yml up -d
```
- > - 如果你想下载并运行特定版本的 RAGFlow slim Docker 镜像，请在 **docker/.env** 文件中找到 `RAGFLOW_IMAGE` 变量，将其改为对应版本。例如 `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.0-slim`，然后再运行上述命令。
+ > - 如果你想下载并运行特定版本的 RAGFlow slim Docker 镜像，请在 **docker/.env** 文件中找到 `RAGFLOW_IMAGE` 变量，将其改为对应版本。例如 `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1-slim`，然后再运行上述命令。
> - 如果您想安装内置 embedding 模型和 Python 库的 dev 版本的 Docker 镜像，需要将 **docker/.env** 文件中的 `RAGFLOW_IMAGE` 变量修改为： `RAGFLOW_IMAGE=infiniflow/ragflow:dev`。
- > - 如果您想安装内置 embedding 模型和 Python 库的指定版本的 RAGFlow Docker 镜像，需要将 **docker/.env** 文件中的 `RAGFLOW_IMAGE` 变量修改为： `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.0`。修改后，再运行上面的命令。
+ > - 如果您想安装内置 embedding 模型和 Python 库的指定版本的 RAGFlow Docker 镜像，需要将 **docker/.env** 文件中的 `RAGFLOW_IMAGE` 变量修改为： `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1`。修改后，再运行上面的命令。
> **注意：** 安装内置 embedding 模型和 Python 库的指定版本的 RAGFlow Docker 镜像大小约 9 GB，可能需要更长时间下载，请耐心等待。
4. 服务器启动成功后再次确认服务器状态：
@@ -296,8 +296,7 @@ docker build -f Dockerfile -t infiniflow/ragflow:dev .
cd web
npm install --force
```
-7. 配置前端，将 **.umirc.ts** 的 `proxy.target` 更新为 `http://127.0.0.1:9380`：
-8. 启动前端服务：
+7. 启动前端服务：
```bash
npm run dev
```
diff --git a/agent/templates/interpreter.json b/agent/templates/interpreter.json
index 7ea83d2dc47..edb3b8be5ef 100644
--- a/agent/templates/interpreter.json
+++ b/agent/templates/interpreter.json
@@ -1,7 +1,7 @@
{
"id": 4,
"title": "Interpreter",
- "description": "Translation Agent: Agentic translation using reflection workflow.\n\nThis is inspired by Andrew NG's project: https://github.com/andrewyng/translation-agent\n\n",
+ "description": "A translation agent based on a reflection agentic workflow, inspired by Andrew Ng's project: https://github.com/andrewyng/translation-agent\n\n1. Prompt an LLM to translate a text into the target language.\n2. Have the LLM reflect on the translation and provide constructive suggestions for improvement.\n3. Use these suggestions to improve the translation.",
"canvas_type": "chatbot",
"dsl": {
"answer": [],
@@ -473,7 +473,7 @@
{
"data": {
"form": {
- "text": "Translation Agent: Agentic translation using reflection workflow\n\nThis is inspired by Andrew NG's project: https://github.com/andrewyng/translation-agent\n\n1. Prompt an LLM to translate a text to target_language;\n2. Have the LLM reflect on the translation to come up with constructive suggestions for improving it;\n3. Use the suggestions to improve the translation."
+ "text": "Translation Agent: Agentic translation using reflection workflow\n\nThis is inspired by Andrew NG's project: https://github.com/andrewyng/translation-agent\n\n1. Prompt an LLM to translate a text into the target language;\n2. Have the LLM reflect on the translation and provide constructive suggestions for improvement;\n3. Use these suggestions to improve the translation."
},
"label": "Note",
"name": "Breif"
diff --git a/agent/templates/seo_blog.json b/agent/templates/seo_blog.json
index f5995f225c7..1be6d6729b5 100644
--- a/agent/templates/seo_blog.json
+++ b/agent/templates/seo_blog.json
@@ -1,7 +1,7 @@
{
"id": 9,
"title": "SEO Blog Generator",
- "description": "A generator for Blogs with SEO just by giving title or keywords.",
+ "description": "A blog generator that creates SEO-optimized content based on your chosen title or keywords.",
"canvas_type": "chatbot",
"dsl": {
"answer": [],
diff --git a/api/apps/chunk_app.py b/api/apps/chunk_app.py
index 4606c8b1d4d..1d21bc103e9 100644
--- a/api/apps/chunk_app.py
+++ b/api/apps/chunk_app.py
@@ -96,7 +96,7 @@ def get():
kb_ids = KnowledgebaseService.get_kb_ids(tenant_id)
chunk = settings.docStoreConn.get(chunk_id, search.index_name(tenant_id), kb_ids)
if chunk is None:
- return server_error_response("Chunk not found")
+ return server_error_response(Exception("Chunk not found"))
k = []
for n in chunk.keys():
if re.search(r"(_vec$|_sm_|_tks|_ltks)", n):
@@ -155,7 +155,7 @@ def set():
v, c = embd_mdl.encode([doc.name, req["content_with_weight"]])
v = 0.1 * v[0] + 0.9 * v[1] if doc.parser_id != ParserType.QA else v[1]
d["q_%d_vec" % len(v)] = v.tolist()
- settings.docStoreConn.insert([d], search.index_name(tenant_id), doc.kb_id)
+ settings.docStoreConn.update({"id": req["chunk_id"]}, d, search.index_name(tenant_id), doc.kb_id)
return get_json_result(data=True)
except Exception as e:
return server_error_response(e)
diff --git a/api/apps/kb_app.py b/api/apps/kb_app.py
index ebac350f10b..c8086aa4209 100644
--- a/api/apps/kb_app.py
+++ b/api/apps/kb_app.py
@@ -29,6 +29,7 @@
from api.utils.api_utils import get_json_result
from api import settings
from rag.nlp import search
+from api.constants import DATASET_NAME_LIMIT
@manager.route('/create', methods=['post'])
@@ -36,10 +37,19 @@
@validate_request("name")
def create():
req = request.json
- req["name"] = req["name"].strip()
- req["name"] = duplicate_name(
+ dataset_name = req["name"]
+ if not isinstance(dataset_name, str):
+ return get_data_error_result(message="Dataset name must be string.")
+ if dataset_name == "":
+ return get_data_error_result(message="Dataset name can't be empty.")
+ if len(dataset_name) >= DATASET_NAME_LIMIT:
+ return get_data_error_result(
+ message=f"Dataset name length is {len(dataset_name)} which is large than {DATASET_NAME_LIMIT}")
+
+ dataset_name = dataset_name.strip()
+ dataset_name = duplicate_name(
KnowledgebaseService.query,
- name=req["name"],
+ name=dataset_name,
tenant_id=current_user.id,
status=StatusEnum.VALID.value)
try:
@@ -73,7 +83,8 @@ def update():
if not KnowledgebaseService.query(
created_by=current_user.id, id=req["kb_id"]):
return get_json_result(
- data=False, message='Only owner of knowledgebase authorized for this operation.', code=settings.RetCode.OPERATING_ERROR)
+ data=False, message='Only owner of knowledgebase authorized for this operation.',
+ code=settings.RetCode.OPERATING_ERROR)
e, kb = KnowledgebaseService.get_by_id(req["kb_id"])
if not e:
@@ -81,7 +92,8 @@ def update():
message="Can't find this knowledgebase!")
if req["name"].lower() != kb.name.lower() \
- and len(KnowledgebaseService.query(name=req["name"], tenant_id=current_user.id, status=StatusEnum.VALID.value)) > 1:
+ and len(
+ KnowledgebaseService.query(name=req["name"], tenant_id=current_user.id, status=StatusEnum.VALID.value)) > 1:
return get_data_error_result(
message="Duplicated knowledgebase name.")
@@ -125,15 +137,16 @@ def detail():
@manager.route('/list', methods=['GET'])
@login_required
def list_kbs():
+ keywords = request.args.get("keywords", "")
page_number = int(request.args.get("page", 1))
items_per_page = int(request.args.get("page_size", 150))
orderby = request.args.get("orderby", "create_time")
desc = request.args.get("desc", True)
try:
tenants = TenantService.get_joined_tenants_by_user_id(current_user.id)
- kbs = KnowledgebaseService.get_by_tenant_ids(
- [m["tenant_id"] for m in tenants], current_user.id, page_number, items_per_page, orderby, desc)
- return get_json_result(data=kbs)
+ kbs, total = KnowledgebaseService.get_by_tenant_ids(
+ [m["tenant_id"] for m in tenants], current_user.id, page_number, items_per_page, orderby, desc, keywords)
+ return get_json_result(data={"kbs": kbs, "total": total})
except Exception as e:
return server_error_response(e)
@@ -151,10 +164,11 @@ def rm():
)
try:
kbs = KnowledgebaseService.query(
- created_by=current_user.id, id=req["kb_id"])
+ created_by=current_user.id, id=req["kb_id"])
if not kbs:
return get_json_result(
- data=False, message='Only owner of knowledgebase authorized for this operation.', code=settings.RetCode.OPERATING_ERROR)
+ data=False, message='Only owner of knowledgebase authorized for this operation.',
+ code=settings.RetCode.OPERATING_ERROR)
for doc in DocumentService.query(kb_id=req["kb_id"]):
if not DocumentService.remove_document(doc, kbs[0].tenant_id):
@@ -168,7 +182,9 @@ def rm():
if not KnowledgebaseService.delete_by_id(req["kb_id"]):
return get_data_error_result(
message="Database error (Knowledgebase removal)!")
- settings.docStoreConn.delete({"kb_id": req["kb_id"]}, search.index_name(kbs[0].tenant_id), req["kb_id"])
+ for kb in kbs:
+ settings.docStoreConn.delete({"kb_id": kb.id}, search.index_name(kb.tenant_id), kb.id)
+ settings.docStoreConn.deleteIdx(search.index_name(kb.tenant_id), kb.id)
return get_json_result(data=True)
except Exception as e:
return server_error_response(e)
diff --git a/api/apps/sdk/session.py b/api/apps/sdk/session.py
index 664c3299ec5..5c04ce0c87a 100644
--- a/api/apps/sdk/session.py
+++ b/api/apps/sdk/session.py
@@ -308,7 +308,6 @@ def rename_field(ans):
if 'docnm_kwd' in chunk_i:
chunk_i['doc_name'] = chunk_i['docnm_kwd']
chunk_i.pop('docnm_kwd')
- conv.message.append(msg[-1])
if not conv.reference:
conv.reference = []
@@ -317,7 +316,6 @@ def rename_field(ans):
final_ans = {"reference": [], "content": ""}
- canvas.messages.append(msg[-1])
canvas.add_user_input(msg[-1]["content"])
if stream:
@@ -415,7 +413,7 @@ def list_session(chat_id,tenant_id):
"document_id": chunk["doc_id"],
"document_name": chunk["docnm_kwd"],
"dataset_id": chunk["kb_id"],
- "image_id": chunk["image_id"],
+ "image_id": chunk.get("image_id", ""),
"similarity": chunk["similarity"],
"vector_similarity": chunk["vector_similarity"],
"term_similarity": chunk["term_similarity"],
diff --git a/api/apps/user_app.py b/api/apps/user_app.py
index 66cae41519c..cc0506316fb 100644
--- a/api/apps/user_app.py
+++ b/api/apps/user_app.py
@@ -252,7 +252,7 @@ def feishu_callback():
if res["code"] != 0:
return redirect("/?error=%s" % res["message"])
- if "contact:user.email:readonly" not in res["data"]["scope"].split(" "):
+ if "contact:user.email:readonly" not in res["data"]["scope"].split():
return redirect("/?error=contact:user.email:readonly not in scope")
session["access_token"] = res["data"]["access_token"]
session["access_token_from"] = "feishu"
diff --git a/api/constants.py b/api/constants.py
index 8d72c7e85aa..e6a97e2c1b1 100644
--- a/api/constants.py
+++ b/api/constants.py
@@ -23,3 +23,5 @@
RAG_FLOW_SERVICE_NAME = "ragflow"
REQUEST_WAIT_SEC = 2
REQUEST_MAX_WAIT_SEC = 300
+
+DATASET_NAME_LIMIT = 128
diff --git a/api/db/services/knowledgebase_service.py b/api/db/services/knowledgebase_service.py
index 1b4c82cf178..47105e7499d 100644
--- a/api/db/services/knowledgebase_service.py
+++ b/api/db/services/knowledgebase_service.py
@@ -16,6 +16,7 @@
from api.db import StatusEnum, TenantPermission
from api.db.db_models import Knowledgebase, DB, Tenant, User, UserTenant,Document
from api.db.services.common_service import CommonService
+from peewee import fn
class KnowledgebaseService(CommonService):
@@ -34,7 +35,7 @@ def list_documents_by_ids(cls,kb_ids):
@classmethod
@DB.connection_context()
def get_by_tenant_ids(cls, joined_tenant_ids, user_id,
- page_number, items_per_page, orderby, desc):
+ page_number, items_per_page, orderby, desc, keywords):
fields = [
cls.model.id,
cls.model.avatar,
@@ -51,20 +52,31 @@ def get_by_tenant_ids(cls, joined_tenant_ids, user_id,
User.avatar.alias('tenant_avatar'),
cls.model.update_time
]
- kbs = cls.model.select(*fields).join(User, on=(cls.model.tenant_id == User.id)).where(
- ((cls.model.tenant_id.in_(joined_tenant_ids) & (cls.model.permission ==
- TenantPermission.TEAM.value)) | (
- cls.model.tenant_id == user_id))
- & (cls.model.status == StatusEnum.VALID.value)
- )
+ if keywords:
+ kbs = cls.model.select(*fields).join(User, on=(cls.model.tenant_id == User.id)).where(
+ ((cls.model.tenant_id.in_(joined_tenant_ids) & (cls.model.permission ==
+ TenantPermission.TEAM.value)) | (
+ cls.model.tenant_id == user_id))
+ & (cls.model.status == StatusEnum.VALID.value),
+ (fn.LOWER(cls.model.name).contains(keywords.lower()))
+ )
+ else:
+ kbs = cls.model.select(*fields).join(User, on=(cls.model.tenant_id == User.id)).where(
+ ((cls.model.tenant_id.in_(joined_tenant_ids) & (cls.model.permission ==
+ TenantPermission.TEAM.value)) | (
+ cls.model.tenant_id == user_id))
+ & (cls.model.status == StatusEnum.VALID.value)
+ )
if desc:
kbs = kbs.order_by(cls.model.getter_by(orderby).desc())
else:
kbs = kbs.order_by(cls.model.getter_by(orderby).asc())
+ count = kbs.count()
+
kbs = kbs.paginate(page_number, items_per_page)
- return list(kbs.dicts())
+ return list(kbs.dicts()), count
@classmethod
@DB.connection_context()
diff --git a/api/db/services/llm_service.py b/api/db/services/llm_service.py
index 1a91c78b632..98d8fd21d01 100644
--- a/api/db/services/llm_service.py
+++ b/api/db/services/llm_service.py
@@ -172,12 +172,17 @@ def increase_usage(cls, tenant_id, llm_type, used_tokens, llm_name=None):
num = 0
try:
- for u in cls.query(tenant_id=tenant_id, llm_name=llm_name):
- num += cls.model.update(used_tokens=u.used_tokens + used_tokens)\
- .where(cls.model.tenant_id == tenant_id, cls.model.llm_name == llm_name)\
+ tenant_llms = cls.query(tenant_id=tenant_id, llm_name=llm_name)
+ if tenant_llms:
+ tenant_llm = tenant_llms[0]
+ num = cls.model.update(used_tokens=tenant_llm.used_tokens + used_tokens)\
+ .where(cls.model.tenant_id == tenant_id, cls.model.llm_factory == tenant_llm.llm_factory, cls.model.llm_name == llm_name)\
.execute()
- except Exception as e:
- pass
+ else:
+ llm_factory = llm_name.split("/")[0] if "/" in llm_name else llm_name
+ num = cls.model.create(tenant_id=tenant_id, llm_factory=llm_factory, llm_name=llm_name, used_tokens=used_tokens)
+ except Exception:
+ logging.exception("TenantLLMService.increase_usage got exception")
return num
@classmethod
diff --git a/api/utils/file_utils.py b/api/utils/file_utils.py
index 5ab61b63bc4..ed014ac0e52 100644
--- a/api/utils/file_utils.py
+++ b/api/utils/file_utils.py
@@ -170,29 +170,52 @@ def filename_type(filename):
return FileType.OTHER.value
def thumbnail_img(filename, blob):
+ """
+ MySQL LongText max length is 65535
+ """
filename = filename.lower()
if re.match(r".*\.pdf$", filename):
pdf = pdfplumber.open(BytesIO(blob))
buffered = BytesIO()
- pdf.pages[0].to_image(resolution=32).annotated.save(buffered, format="png")
- return buffered.getvalue()
+ resolution = 32
+ img = None
+ for _ in range(10):
+ # https://github.com/jsvine/pdfplumber?tab=readme-ov-file#creating-a-pageimage-with-to_image
+ pdf.pages[0].to_image(resolution=resolution).annotated.save(buffered, format="png")
+ img = buffered.getvalue()
+ if len(img) >= 64000 and resolution >= 2:
+ resolution = resolution / 2
+ buffered = BytesIO()
+ else:
+ break
+ return img
- if re.match(r".*\.(jpg|jpeg|png|tif|gif|icon|ico|webp)$", filename):
+ elif re.match(r".*\.(jpg|jpeg|png|tif|gif|icon|ico|webp)$", filename):
image = Image.open(BytesIO(blob))
image.thumbnail((30, 30))
buffered = BytesIO()
image.save(buffered, format="png")
return buffered.getvalue()
- if re.match(r".*\.(ppt|pptx)$", filename):
+ elif re.match(r".*\.(ppt|pptx)$", filename):
import aspose.slides as slides
import aspose.pydrawing as drawing
try:
with slides.Presentation(BytesIO(blob)) as presentation:
buffered = BytesIO()
- presentation.slides[0].get_thumbnail(0.03, 0.03).save(
- buffered, drawing.imaging.ImageFormat.png)
- return buffered.getvalue()
+ scale = 0.03
+ img = None
+ for _ in range(10):
+ # https://reference.aspose.com/slides/python-net/aspose.slides/slide/get_thumbnail/#float-float
+ presentation.slides[0].get_thumbnail(scale, scale).save(
+ buffered, drawing.imaging.ImageFormat.png)
+ img = buffered.getvalue()
+ if len(img) >= 64000:
+ scale = scale / 2.0
+ buffered = BytesIO()
+ else:
+ break
+ return img
except Exception:
pass
return None
diff --git a/deepdoc/parser/docx_parser.py b/deepdoc/parser/docx_parser.py
index 1c1c14d3041..1910f4380c4 100644
--- a/deepdoc/parser/docx_parser.py
+++ b/deepdoc/parser/docx_parser.py
@@ -47,7 +47,7 @@ def blockType(b):
for p, n in patt:
if re.search(p, b):
return n
- tks = [t for t in rag_tokenizer.tokenize(b).split(" ") if len(t) > 1]
+ tks = [t for t in rag_tokenizer.tokenize(b).split() if len(t) > 1]
if len(tks) > 3:
if len(tks) < 12:
return "Tx"
diff --git a/deepdoc/parser/pdf_parser.py b/deepdoc/parser/pdf_parser.py
index 331d5da1726..9c6139469d5 100644
--- a/deepdoc/parser/pdf_parser.py
+++ b/deepdoc/parser/pdf_parser.py
@@ -108,13 +108,13 @@ def _updown_concat_features(self, up, down):
h = max(self.__height(up), self.__height(down))
y_dis = self._y_dis(up, down)
LEN = 6
- tks_down = rag_tokenizer.tokenize(down["text"][:LEN]).split(" ")
- tks_up = rag_tokenizer.tokenize(up["text"][-LEN:]).split(" ")
+ tks_down = rag_tokenizer.tokenize(down["text"][:LEN]).split()
+ tks_up = rag_tokenizer.tokenize(up["text"][-LEN:]).split()
tks_all = up["text"][-LEN:].strip() \
+ (" " if re.match(r"[a-zA-Z0-9]+",
up["text"][-1] + down["text"][0]) else "") \
+ down["text"][:LEN].strip()
- tks_all = rag_tokenizer.tokenize(tks_all).split(" ")
+ tks_all = rag_tokenizer.tokenize(tks_all).split()
fea = [
up.get("R", -1) == down.get("R", -1),
y_dis / h,
@@ -565,13 +565,13 @@ def _filter_forpages(self):
if i >= len(self.boxes):
break
prefix = self.boxes[i]["text"].strip()[:3] if not eng else " ".join(
- self.boxes[i]["text"].strip().split(" ")[:2])
+ self.boxes[i]["text"].strip().split()[:2])
while not prefix:
self.boxes.pop(i)
if i >= len(self.boxes):
break
prefix = self.boxes[i]["text"].strip()[:3] if not eng else " ".join(
- self.boxes[i]["text"].strip().split(" ")[:2])
+ self.boxes[i]["text"].strip().split()[:2])
self.boxes.pop(i)
if i >= len(self.boxes) or not prefix:
break
diff --git a/deepdoc/parser/resume/entities/corporations.py b/deepdoc/parser/resume/entities/corporations.py
index c26f58aebd4..142b0f5e492 100644
--- a/deepdoc/parser/resume/entities/corporations.py
+++ b/deepdoc/parser/resume/entities/corporations.py
@@ -47,7 +47,7 @@ def corpNorm(nm, add_region=True):
nm = re.sub(r"(计算机|技术|(技术|科技|网络)*有限公司|公司|有限|研发中心|中国|总部)$", "", nm, 10000, re.IGNORECASE)
if not nm or (len(nm)<5 and not regions.isName(nm[0:2])):return nm
- tks = rag_tokenizer.tokenize(nm).split(" ")
+ tks = rag_tokenizer.tokenize(nm).split()
reg = [t for i,t in enumerate(tks) if regions.isName(t) and (t != "中国" or i > 0)]
nm = ""
for t in tks:
diff --git a/deepdoc/parser/resume/entities/schools.py b/deepdoc/parser/resume/entities/schools.py
index 598d7ae83f5..31662cde94f 100644
--- a/deepdoc/parser/resume/entities/schools.py
+++ b/deepdoc/parser/resume/entities/schools.py
@@ -44,7 +44,7 @@ def loadRank(fnm):
def split(txt):
tks = []
- for t in re.sub(r"[ \t]+", " ",txt).split(" "):
+ for t in re.sub(r"[ \t]+", " ",txt).split():
if tks and re.match(r".*[a-zA-Z]$", tks[-1]) and \
re.match(r"[a-zA-Z]", t) and tks:
tks[-1] = tks[-1] + " " + t
diff --git a/deepdoc/parser/resume/step_one.py b/deepdoc/parser/resume/step_one.py
index 90e52e45078..96cc668d316 100644
--- a/deepdoc/parser/resume/step_one.py
+++ b/deepdoc/parser/resume/step_one.py
@@ -80,7 +80,7 @@ def deal_obj(obj, k, kk):
def loadjson(line):
try:
return json.loads(line)
- except Exception as e:
+ except Exception:
pass
return {}
@@ -183,4 +183,4 @@ def arr2str(a):
"\r",
"\\n"))
# print(df.values.tolist())
- return dict(zip([n.split(" ")[0] for n in FIELDS], df.values.tolist()[0]))
+ return dict(zip([n.split()[0] for n in FIELDS], df.values.tolist()[0]))
diff --git a/deepdoc/parser/resume/step_two.py b/deepdoc/parser/resume/step_two.py
index afc5fb47da7..7d429777423 100644
--- a/deepdoc/parser/resume/step_two.py
+++ b/deepdoc/parser/resume/step_two.py
@@ -100,7 +100,7 @@ def forEdu(cv):
if n.get("school_name") and isinstance(n["school_name"], str):
sch.append(re.sub(r"(211|985|重点大学|[,&;；-])", "", n["school_name"]))
e["sch_nm_kwd"] = sch[-1]
- fea.append(rag_tokenizer.fine_grained_tokenize(rag_tokenizer.tokenize(n.get("school_name", ""))).split(" ")[-1])
+ fea.append(rag_tokenizer.fine_grained_tokenize(rag_tokenizer.tokenize(n.get("school_name", ""))).split()[-1])
if n.get("discipline_name") and isinstance(n["discipline_name"], str):
maj.append(n["discipline_name"])
@@ -485,7 +485,7 @@ def hasValues(flds):
nm = re.sub(r"[\n——\-\(（\+].*", "", cv["name"].strip())
nm = re.sub(r"[ \t ]+", " ", nm)
if re.match(r"[a-zA-Z ]+$", nm):
- if len(nm.split(" ")) > 1:
+ if len(nm.split()) > 1:
cv["name"] = nm
else:
nm = ""
@@ -503,7 +503,7 @@ def hasValues(flds):
for py in PY.get_pinyins(nm[:20], ''):
for i in range(2, len(py) + 1): cv["name_py_pref_tks"] += " " + py[:i]
for py in PY.get_pinyins(nm[:20], ' '):
- py = py.split(" ")
+ py = py.split()
for i in range(1, len(py) + 1): cv["name_py_pref0_tks"] += " " + "".join(py[:i])
cv["name_kwd"] = name
diff --git a/deepdoc/vision/table_structure_recognizer.py b/deepdoc/vision/table_structure_recognizer.py
index 5759c0f644b..be48ca95e4c 100644
--- a/deepdoc/vision/table_structure_recognizer.py
+++ b/deepdoc/vision/table_structure_recognizer.py
@@ -117,7 +117,7 @@ def blockType(b):
for p, n in patt:
if re.search(p, b["text"].strip()):
return n
- tks = [t for t in rag_tokenizer.tokenize(b["text"]).split(" ") if len(t) > 1]
+ tks = [t for t in rag_tokenizer.tokenize(b["text"]).split() if len(t) > 1]
if len(tks) > 3:
if len(tks) < 12:
return "Tx"
diff --git a/docker/.env b/docker/.env
index 6f8ce6a4c59..d64cd2be779 100644
--- a/docker/.env
+++ b/docker/.env
@@ -22,7 +22,6 @@ ES_HOST=es01
ES_PORT=1200
# The password for Elasticsearch.
-# When updated, you must revise the `es.password` entry in service_conf.yaml accordingly.
ELASTIC_PASSWORD=infini_rag_flow
# The port used to expose the Kibana service to the host machine,
@@ -44,7 +43,6 @@ INFINITY_HTTP_PORT=23820
INFINITY_PSQL_PORT=5432
# The password for MySQL.
-# When updated, you must revise the `mysql.password` entry in service_conf.yaml.
MYSQL_PASSWORD=infini_rag_flow
# The hostname where the MySQL service is exposed
MYSQL_HOST=mysql
@@ -75,7 +73,6 @@ REDIS_HOST=redis
# allowing EXTERNAL access to the Redis service running inside the Docker container.
REDIS_PORT=6379
# The password for Redis.
-# When updated, you must revise the `redis.password` entry in service_conf.yaml accordingly.
REDIS_PASSWORD=infini_rag_flow
# The port used to expose RAGFlow's HTTP API service to the host machine,
@@ -118,7 +115,6 @@ RAGFLOW_IMAGE=infiniflow/ragflow:dev-slim
# RAGFLOW_IMAGE=swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow:dev
# RAGFLOW_IMAGE=registry.cn-hangzhou.aliyuncs.com/infiniflow/ragflow:dev
-
# The local time zone.
TIMEZONE='Asia/Shanghai'
@@ -130,6 +126,6 @@ TIMEZONE='Asia/Shanghai'
# MACOS=1
# The maximum file size for each uploaded file, in bytes.
-# You can uncomment this line and update the value if you wish to change 128M file size limit
+# You can uncomment this line and update the value if you wish to change the 128M file size limit
# MAX_CONTENT_LENGTH=134217728
diff --git a/docker/README.md b/docker/README.md
index bab35dbe9e7..da51bbf6d36 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -133,14 +133,14 @@ The [.env](./.env) file contains important environment variables for Docker.
- `mysql`
- `name`: The MySQL database name. Defaults to `rag_flow`.
- `user`: The username for MySQL.
- - `password`: The password for MySQL. When updated, you must revise the `MYSQL_PASSWORD` variable in [.env](./.env) accordingly.
+ - `password`: The password for MySQL.
- `port`: The MySQL serving port inside the Docker container. Defaults to `3306`.
- `max_connections`: The maximum number of concurrent connections to the MySQL database. Defaults to `100`.
- `stale_timeout`: Timeout in seconds.
- `minio`
- - `user`: The username for MinIO. When updated, you must revise the `MINIO_USER` variable in [.env](./.env) accordingly.
- - `password`: The password for MinIO. When updated, you must revise the `MINIO_PASSWORD` variable in [.env](./.env) accordingly.
+ - `user`: The username for MinIO.
+ - `password`: The password for MinIO.
- `host`: The MinIO serving IP *and* port inside the Docker container. Defaults to `minio:9000`.
- `oauth`
diff --git a/docker/docker-compose-base.yml b/docker/docker-compose-base.yml
index 2988bae6b1c..48cfda0600f 100644
--- a/docker/docker-compose-base.yml
+++ b/docker/docker-compose-base.yml
@@ -39,6 +39,8 @@ services:
image: infiniflow/infinity:v0.5.0-dev5
volumes:
- infinity_data:/var/infinity
+ - ./infinity_conf.toml:/infinity_conf.toml
+ command: ["-f", "/infinity_conf.toml"]
ports:
- ${INFINITY_THRIFT_PORT}:23817
- ${INFINITY_HTTP_PORT}:23820
diff --git a/docker/docker-compose-gpu.yml b/docker/docker-compose-gpu.yml
index 1f0fe8a4c85..fc82f03f498 100644
--- a/docker/docker-compose-gpu.yml
+++ b/docker/docker-compose-gpu.yml
@@ -1,14 +1,11 @@
include:
- - path: ./docker-compose-base.yml
- env_file: ./.env
+ - ./docker-compose-base.yml
services:
ragflow:
depends_on:
mysql:
condition: service_healthy
- es01:
- condition: service_healthy
image: ${RAGFLOW_IMAGE}
container_name: ragflow-server
ports:
@@ -20,13 +17,18 @@ services:
- ./nginx/ragflow.conf:/etc/nginx/conf.d/ragflow.conf
- ./nginx/proxy.conf:/etc/nginx/proxy.conf
- ./nginx/nginx.conf:/etc/nginx/nginx.conf
+ env_file: .env
environment:
- TZ=${TIMEZONE}
- - HF_ENDPOINT=https://huggingface.co
+ - HF_ENDPOINT=${HF_ENDPOINT}
- MACOS=${MACOS}
networks:
- ragflow
- restart: always
+ restart: on-failure
+ # https://docs.docker.com/engine/daemon/prometheus/#create-a-prometheus-configuration
+ # If you're using Docker Desktop, the --add-host flag is optional. This flag makes sure that the host's internal IP gets exposed to the Prometheus container.
+ extra_hosts:
+ - "host.docker.internal:host-gateway"
deploy:
resources:
reservations:
diff --git a/docker/infinity_conf.toml b/docker/infinity_conf.toml
new file mode 100644
index 00000000000..cd0a886ac5c
--- /dev/null
+++ b/docker/infinity_conf.toml
@@ -0,0 +1,66 @@
+[general]
+version = "0.5.0"
+time_zone = "utc-8"
+
+[network]
+server_address = "0.0.0.0"
+postgres_port = 5432
+http_port = 23820
+client_port = 23817
+connection_pool_size = 128
+
+[log]
+log_filename = "infinity.log"
+log_dir = "/var/infinity/log"
+log_to_stdout = true
+log_file_max_size = "100MB"
+log_file_rotate_count = 10
+
+# trace/debug/info/warning/error/critical 6 log levels, default: info
+log_level = "info"
+
+[storage]
+persistence_dir = "/var/infinity/persistence"
+data_dir = "/var/infinity/data"
+# periodically activates garbage collection:
+# 0 means real-time,
+# s means seconds, for example "60s", 60 seconds
+# m means minutes, for example "60m", 60 minutes
+# h means hours, for example "1h", 1 hour
+optimize_interval = "10s"
+cleanup_interval = "60s"
+compact_interval = "120s"
+storage_type = "local"
+
+# dump memory index entry when it reachs the capacity
+mem_index_capacity = 65536
+
+# S3 storage config example:
+# [storage.object_storage]
+# url = "127.0.0.1:9000"
+# bucket_name = "infinity"
+# access_key = "minioadmin"
+# secret_key = "minioadmin"
+# enable_https = false
+
+[buffer]
+buffer_manager_size = "8GB"
+lru_num = 7
+temp_dir = "/var/infinity/tmp"
+result_cache = "off"
+memindex_memory_quota = "4GB"
+
+[wal]
+wal_dir = "/var/infinity/wal"
+full_checkpoint_interval = "30s"
+delta_checkpoint_interval = "5s"
+# delta_checkpoint_threshold = 1000000000
+wal_compact_threshold = "1GB"
+
+# flush_at_once: write and flush log each commit
+# only_write: write log, OS control when to flush the log, default
+# flush_per_second: logs are written after each commit and flushed to disk per second.
+wal_flush = "only_write"
+
+[resource]
+resource_dir = "/var/infinity/resource"
diff --git a/docs/guides/configure_knowledge_base.md b/docs/guides/configure_knowledge_base.md
index aef6ad5165e..457062b34c5 100644
--- a/docs/guides/configure_knowledge_base.md
+++ b/docs/guides/configure_knowledge_base.md
@@ -103,7 +103,7 @@ RAGFlow features visibility and explainability, allowing you to view the chunkin
2. Hover over each snapshot for a quick view of each chunk.
-3. Double click the chunked texts to add keywords or make *manual* changes where necessary:
+3. Double-click the chunked texts to add keywords or make *manual* changes where necessary:
@@ -111,7 +111,7 @@ RAGFlow features visibility and explainability, allowing you to view the chunkin
You can add keywords to a file chunk to increase its ranking for queries containing those keywords. This action increases its keyword weight and can improve its position in search list.
:::
-4. In Retrieval testing, ask a quick question in **Test text** to double check if your configurations work:
+4. In Retrieval testing, ask a quick question in **Test text** to double-check if your configurations work:
_As you can tell from the following, RAGFlow responds with truthful citations._
@@ -128,7 +128,7 @@ RAGFlow uses multiple recall of both full-text search and vector search in its c
## Search for knowledge base
-As of RAGFlow v0.14.0, the search feature is still in a rudimentary form, supporting only knowledge base search by name.
+As of RAGFlow v0.14.1, the search feature is still in a rudimentary form, supporting only knowledge base search by name.
diff --git a/docs/guides/deploy_local_llm.mdx b/docs/guides/deploy_local_llm.mdx
index 5dcaefc5a09..1c7b856d476 100644
--- a/docs/guides/deploy_local_llm.mdx
+++ b/docs/guides/deploy_local_llm.mdx
@@ -74,9 +74,9 @@ In the popup window, complete basic settings for Ollama:
4. OPTIONAL: Switch on the toggle under **Does it support Vision?** if your model includes an image-to-text model.
:::caution NOTE
+- If RAGFlow is in Docker and Ollama runs on the same host machine, use `http://host.docker.internal:11434` as base URL.
- If your Ollama and RAGFlow run on the same machine, use `http://localhost:11434` as base URL.
-- If your Ollama and RAGFlow run on the same machine and Ollama is in Docker, use `http://host.docker.internal:11434` as base URL.
-- If your Ollama runs on a different machine from RAGFlow, use `http://vm.max_map_count ≥ 262144:{t('chunk.keyword')} *
+{t('chunk.keyword')} *