Merge branch 'main' of https://github.com/infiniflow/ragflow

isthaison · Dec 4, 2024 · db9fb20 · db9fb20
2 parents cde23ad + f3318b2
commit db9fb20
Show file tree

Hide file tree

Showing 37 changed files with 632 additions and 115 deletions.
diff --git a/README.md b/README.md
@@ -75,9 +75,9 @@ Try our demo at [https://demo.ragflow.io](https://demo.ragflow.io).
 
 ## 🔥 Latest Updates
 
+- 2024-12-04 Adds support for pagerank score in knowledge base.
 - 2024-11-22 Adds more variables to Agent.
 - 2024-11-01 Adds keyword extraction and related question generation to the parsed chunks to improve the accuracy of retrieval.
-- 2024-09-13 Adds search mode for knowledge base Q&A.
 - 2024-08-22 Support text to SQL statements through RAG.
 - 2024-08-02 Supports GraphRAG inspired by [graphrag](https://github.com/microsoft/graphrag) and mind map.
 

diff --git a/README_id.md b/README_id.md
@@ -72,9 +72,9 @@ Coba demo kami di [https://demo.ragflow.io](https://demo.ragflow.io).
 
 ## 🔥 Pembaruan Terbaru
 
-- 22-11-2024 Peningkatan definisi dan penggunaan variabel di Agen.
+- 2024-12-04 Mendukung skor pagerank ke basis pengetahuan.
+- 2024-11-22 Peningkatan definisi dan penggunaan variabel di Agen.
 - 2024-11-01: Penambahan ekstraksi kata kunci dan pembuatan pertanyaan terkait untuk meningkatkan akurasi pengambilan.
-- 2024-09-13: Penambahan mode pencarian untuk Q&A basis pengetahuan.
 - 2024-08-22: Dukungan untuk teks ke pernyataan SQL melalui RAG.
 - 2024-08-02: Dukungan GraphRAG yang terinspirasi oleh [graphrag](https://github.com/microsoft/graphrag) dan mind map.
 

diff --git a/README_ja.md b/README_ja.md
@@ -54,9 +54,9 @@
 
 ## 🔥 最新情報
 
+- 2024-12-04 ナレッジ ベースへのページランク スコアをサポートしました。
 - 2024-11-22 エージェントでの変数の定義と使用法を改善しました。
 - 2024-11-01 再現の精度を向上させるために、解析されたチャンクにキーワード抽出と関連質問の生成を追加しました。
-- 2024-09-13 ナレッジベース Q&A の検索モードを追加しました。
 - 2024-08-22 RAG を介して SQL ステートメントへのテキストをサポートします。
 - 2024-08-02 [graphrag](https://github.com/microsoft/graphrag) からインスピレーションを得た GraphRAG とマインド マップをサポートします。
 

diff --git a/README_ko.md b/README_ko.md
@@ -56,12 +56,12 @@
 
 ## 🔥 업데이트
 
+- 2024-12-04 지식베이스에 대한 페이지랭크 점수를 지원합니다.
+
 - 2024-11-22 에이전트의 변수 정의 및 사용을 개선했습니다.
 
 - 2024-11-01 파싱된 청크에 키워드 추출 및 관련 질문 생성을 추가하여 재현율을 향상시킵니다.
 
-- 2024-09-13 지식베이스 Q&A 검색 모드를 추가합니다.
-
 - 2024-08-22 RAG를 통해 SQL 문에 텍스트를 지원합니다.
 
 - 2024-08-02: [graphrag](https://github.com/microsoft/graphrag)와 마인드맵에서 영감을 받은 GraphRAG를 지원합니다.

diff --git a/README_zh.md b/README_zh.md
@@ -54,9 +54,9 @@
 
 ## 🔥 近期更新
 
+- 2024-12-04 支持知识库的 Pagerank 分数。
 - 2024-11-22 完善了 Agent 中的变量定义和使用。
 - 2024-11-01 对解析后的 chunk 加入关键词抽取和相关问题生成以提高召回的准确度。
-- 2024-09-13 增加知识库问答搜索模式。
 - 2024-08-22 支持用 RAG 技术实现从自然语言到 SQL 语句的转换。
 - 2024-08-02 支持 GraphRAG 启发于 [graphrag](https://github.com/microsoft/graphrag) 和思维导图。
 
@@ -332,9 +332,9 @@ RAGFlow 只有通过开源协作才能蓬勃发展。秉持这一精神,我们
 
 ## 👥 加入社区
 
-扫二维码添加 InfinifFlow 小助手，进 RAGFlow 交流群。
+扫二维码添加 RAGFlow 小助手，进 RAGFlow 交流群。
 
 <p align="center">
-  <img src="https://github.com/user-attachments/assets/87095713-7ad2-4c48-bd11-10030d0e30ae" width=50% height=50%>
+  <img src="https://github.com/infiniflow/ragflow/assets/7248/bccf284f-46f2-4445-9809-8f1030fb7585" width=50% height=50%>
 </p>
 
diff --git a/agent/component/__init__.py b/agent/component/__init__.py
@@ -31,6 +31,8 @@
 from .crawler import Crawler, CrawlerParam
 from .invoke import Invoke, InvokeParam
 from .template import Template, TemplateParam
+from .email import Email, EmailParam
+
 
 
 def component_class(class_name):

diff --git a/agent/component/email.py b/agent/component/email.py
@@ -0,0 +1,138 @@
+#
+#  Copyright 2024 The InfiniFlow Authors. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from abc import ABC
+import json
+import smtplib
+import logging
+from email.mime.text import MIMEText
+from email.mime.multipart import MIMEMultipart
+from email.header import Header
+from email.utils import formataddr
+from agent.component.base import ComponentBase, ComponentParamBase
+
+class EmailParam(ComponentParamBase):
+    """
+    Define the Email component parameters.
+    """
+    def __init__(self):
+        super().__init__()
+        # Fixed configuration parameters
+        self.smtp_server = ""  # SMTP server address
+        self.smtp_port = 465  # SMTP port
+        self.email = ""  # Sender email
+        self.password = ""  # Email authorization code
+        self.sender_name = ""  # Sender name
+
+    def check(self):
+        # Check required parameters
+        self.check_empty(self.smtp_server, "SMTP Server")
+        self.check_empty(self.email, "Email")
+        self.check_empty(self.password, "Password")
+        self.check_empty(self.sender_name, "Sender Name")
+
+class Email(ComponentBase, ABC):
+    component_name = "Email"
+
+    def _run(self, history, **kwargs):
+        # Get upstream component output and parse JSON
+        ans = self.get_input()
+        content = "".join(ans["content"]) if "content" in ans else ""
+        if not content:
+            return Email.be_output("No content to send")
+
+        success = False
+        try:
+            # Parse JSON string passed from upstream
+            email_data = json.loads(content)
+
+            # Validate required fields
+            if "to_email" not in email_data:
+                return Email.be_output("Missing required field: to_email")
+
+            # Create email object
+            msg = MIMEMultipart('alternative')
+
+            # Properly handle sender name encoding
+            msg['From'] = formataddr((str(Header(self._param.sender_name,'utf-8')), self._param.email))
+            msg['To'] = email_data["to_email"]
+            if "cc_email" in email_data and email_data["cc_email"]:
+                msg['Cc'] = email_data["cc_email"]
+            msg['Subject'] = Header(email_data.get("subject", "No Subject"), 'utf-8').encode()
+
+            # Use content from email_data or default content
+            email_content = email_data.get("content", "No content provided")
+            # msg.attach(MIMEText(email_content, 'plain', 'utf-8'))
+            msg.attach(MIMEText(email_content, 'html', 'utf-8'))
+
+            # Connect to SMTP server and send
+            logging.info(f"Connecting to SMTP server {self._param.smtp_server}:{self._param.smtp_port}")
+
+            context = smtplib.ssl.create_default_context()
+            with smtplib.SMTP_SSL(self._param.smtp_server, self._param.smtp_port, context=context) as server:
+                # Login
+                logging.info(f"Attempting to login with email: {self._param.email}")
+                server.login(self._param.email, self._param.password)
+
+                # Get all recipient list
+                recipients = [email_data["to_email"]]
+                if "cc_email" in email_data and email_data["cc_email"]:
+                    recipients.extend(email_data["cc_email"].split(','))
+
+                # Send email
+                logging.info(f"Sending email to recipients: {recipients}")
+                try:
+                    server.send_message(msg, self._param.email, recipients)
+                    success = True
+                except Exception as e:
+                    logging.error(f"Error during send_message: {str(e)}")
+                    # Try alternative method
+                    server.sendmail(self._param.email, recipients, msg.as_string())
+                    success = True
+
+                try:
+                    server.quit()
+                except Exception as e:
+                    # Ignore errors when closing connection
+                    logging.warning(f"Non-fatal error during connection close: {str(e)}")
+
+            if success:
+                return Email.be_output("Email sent successfully")
+
+        except json.JSONDecodeError:
+            error_msg = "Invalid JSON format in input"
+            logging.error(error_msg)
+            return Email.be_output(error_msg)
+
+        except smtplib.SMTPAuthenticationError:
+            error_msg = "SMTP Authentication failed. Please check your email and authorization code."
+            logging.error(error_msg)
+            return Email.be_output(f"Failed to send email: {error_msg}")
+
+        except smtplib.SMTPConnectError:
+            error_msg = f"Failed to connect to SMTP server {self._param.smtp_server}:{self._param.smtp_port}"
+            logging.error(error_msg)
+            return Email.be_output(f"Failed to send email: {error_msg}")
+
+        except smtplib.SMTPException as e:
+            error_msg = f"SMTP error occurred: {str(e)}"
+            logging.error(error_msg)
+            return Email.be_output(f"Failed to send email: {error_msg}")
+
+        except Exception as e:
+            error_msg = f"Unexpected error: {str(e)}"
+            logging.error(error_msg)
+            return Email.be_output(f"Failed to send email: {error_msg}") 
diff --git a/api/apps/file_app.py b/api/apps/file_app.py
@@ -323,14 +323,19 @@ def rename():
 
 
 @manager.route('/get/<file_id>', methods=['GET'])
-# @login_required
+@login_required
 def get(file_id):
     try:
         e, file = FileService.get_by_id(file_id)
         if not e:
             return get_data_error_result(message="Document not found!")
-        b, n = File2DocumentService.get_storage_address(file_id=file_id)
-        response = flask.make_response(STORAGE_IMPL.get(b, n))
+
+        blob = STORAGE_IMPL.get(file.parent_id, file.location)
+        if not blob:
+            b, n = File2DocumentService.get_storage_address(file_id=file_id)
+            blob = STORAGE_IMPL.get(b, n)
+
+        response = flask.make_response(blob)
         ext = re.search(r"\.([^.]+)$", file.name)
         if ext:
             if file.type == FileType.VISUAL.value:

diff --git a/rag/llm/rerank_model.py b/rag/llm/rerank_model.py
@@ -126,7 +126,6 @@ def __init__(self, key=None, model_name="maidalun1020/bce-reranker-base_v1", **k
             with YoudaoRerank._model_lock:
                 if not YoudaoRerank._model:
                     try:
-                        logging.info("LOADING BCE...")
                         YoudaoRerank._model = RerankerModel(model_name_or_path=os.path.join(
                             get_home_cache_dir(),
                             re.sub(r"^[a-zA-Z0-9]+/", "", model_name)))

diff --git a/web/package.json b/web/package.json
@@ -20,6 +20,7 @@
     "@ant-design/icons": "^5.2.6",
     "@ant-design/pro-components": "^2.6.46",
     "@ant-design/pro-layout": "^7.17.16",
+    "@antv/g": "^6.1.11",
     "@antv/g6": "^5.0.10",
     "@hookform/resolvers": "^3.9.1",
     "@js-preview/excel": "^1.7.8",
@@ -72,6 +73,7 @@
     "react-i18next": "^14.0.0",
     "react-infinite-scroll-component": "^6.1.0",
     "react-markdown": "^9.0.1",
+    "react-pdf": "^9.1.1",
     "react-pdf-highlighter": "^6.1.0",
     "react-string-replace": "^1.1.1",
     "react-syntax-highlighter": "^15.5.0",

diff --git a/web/src/assets/svg/email.svg b/web/src/assets/svg/email.svg
diff --git a/web/src/hooks/file-manager-hooks.ts b/web/src/hooks/file-manager-hooks.ts
@@ -2,6 +2,7 @@ import { ResponseType } from '@/interfaces/database/base';
 import { IFolder } from '@/interfaces/database/file-manager';
 import { IConnectRequestBody } from '@/interfaces/request/file-manager';
 import fileManagerService from '@/services/file-manager-service';
+import { downloadFileFromBlob } from '@/utils/file-util';
 import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
 import { PaginationProps, UploadFile, message } from 'antd';
 import React, { useCallback } from 'react';
@@ -114,6 +115,39 @@ export const useDeleteFile = () => {
   return { data, loading, deleteFile: mutateAsync };
 };
 
+export const useDownloadFile = () => {
+  const {
+    data,
+    isPending: loading,
+    mutateAsync,
+  } = useMutation({
+    mutationKey: ['downloadFile'],
+    mutationFn: async (params: { id: string; filename?: string }) => {
+      const response = await fileManagerService.getFile({}, params.id);
+      const blob = new Blob([response.data], { type: response.data.type });
+      downloadFileFromBlob(blob, params.filename);
+    },
+  });
+  return { data, loading, downloadFile: mutateAsync };
+};
+
+export const useLoadFile = () => {
+  const {
+    data,
+    isPending: loading,
+    mutateAsync,
+    error,
+  } = useMutation({
+    mutationKey: ['downloadFile'],
+    mutationFn: async (params: { id: string }) => {
+      const response = await fileManagerService.getFile({}, params.id);
+      const blob = new Blob([response.data], { type: response.data.type });
+      return blob;
+    },
+  });
+  return { data, loading, loadFile: mutateAsync, error };
+};
+
 export const useRenameFile = () => {
   const queryClient = useQueryClient();
   const { t } = useTranslation();

diff --git a/web/src/locales/en.ts b/web/src/locales/en.ts
@@ -1050,6 +1050,31 @@ The above is the content you need to summarize.`,
       template: 'Template',
       templateDescription:
         'This component is used for typesetting the outputs of various components.',
+      emailComponent: 'Email',
+      emailDescription: 'Send email to specified address',
+      smtpServer: 'SMTP Server',
+      smtpPort: 'SMTP Port',
+      senderEmail: 'Sender Email',
+      authCode: 'Authorization Code',
+      senderName: 'Sender Name',
+      toEmail: 'Recipient Email',
+      ccEmail: 'CC Email',
+      emailSubject: 'Subject',
+      emailContent: 'Content',
+      smtpServerRequired: 'Please input SMTP server address',
+      senderEmailRequired: 'Please input sender email',
+      authCodeRequired: 'Please input authorization code',
+      toEmailRequired: 'Please input recipient email',
+      emailContentRequired: 'Please input email content',
+      emailSentSuccess: 'Email sent successfully',
+      emailSentFailed: 'Failed to send email',
+      dynamicParameters: 'Dynamic Parameters',
+      jsonFormatTip:
+        'Upstream component should provide JSON string in following format:',
+      toEmailTip: 'to_email: Recipient email (Required)',
+      ccEmailTip: 'cc_email: CC email (Optional)',
+      subjectTip: 'subject: Email subject (Optional)',
+      contentTip: 'content: Email content (Optional)',
     },
     footer: {
       profile: 'All rights reserved @ React',

diff --git a/web/src/locales/zh.ts b/web/src/locales/zh.ts
@@ -1029,6 +1029,30 @@ export default {
       testRun: '试运行',
       template: '模板转换',
       templateDescription: '该组件用于排版各种组件的输出。',
+      emailComponent: '邮件',
+      emailDescription: '发送邮件到指定邮箱',
+      smtpServer: 'SMTP服务器',
+      smtpPort: 'SMTP端口',
+      senderEmail: '发件人邮箱',
+      authCode: '授权码',
+      senderName: '发件人名称',
+      toEmail: '收件人邮箱',
+      ccEmail: '抄送邮箱',
+      emailSubject: '邮件主题',
+      emailContent: '邮件内容',
+      smtpServerRequired: '请输入SMTP服务器地址',
+      senderEmailRequired: '请输入发件人邮箱',
+      authCodeRequired: '请输入授权码',
+      toEmailRequired: '请输入收件人邮箱',
+      emailContentRequired: '请输入邮件内容',
+      emailSentSuccess: '邮件发送成功',
+      emailSentFailed: '邮件发送失败',
+      dynamicParameters: '动态参数说明',
+      jsonFormatTip: '上游组件需要传入以下格式的JSON字符串:',
+      toEmailTip: 'to_email: 收件人邮箱(必填)',
+      ccEmailTip: 'cc_email: 抄送邮箱(可选)',
+      subjectTip: 'subject: 邮件主题(可选)',
+      contentTip: 'content: 邮件内容(可选)',
     },
     footer: {
       profile: 'All rights reserved @ React',