Merge pull request #162 from alipay/dev

feat: Version 0.0.13 Release

CHANGELOG.md

@@ -24,6 +24,27 @@ Note - Additional remarks regarding the version.
 ***************************************************
 
 # Version Update History
+# [0.0.13] - 2024-09-12
+### Added
+- RAG(Retrieval-Augmented Generation) Component Version Update
+- This version provides a standard operating procedure for knowledge base construction and the RAG retrieval recall stage. The component covers a series of RAG atomic capabilities, including data loading, data processing, index construction, knowledge storage, intent rewriting, and retrieval re-ranking, helping users to quickly build a general RAG intelligent agent solution in open-source scenarios.
+
+- Intelligent Agent Product Platform Update
+- This version introduces new capabilities such as intelligent agent canvas orchestration, private knowledge base construction, and custom plugin support, enabling users to quickly build and orchestrate intelligent agents through a low-code, visual approach.
+
+- Added GLM Default Model Component for Zhipu
+- Added SQLiteStore Storage Component
+- Added Flow Orchestration Execution Engine
+
+### Note 
+- Default path optimization for system_db_uri
+The default path is already compatible with the Windows platform, for more details, please refer to [issue142](https://github.com/alipay/agentUniverse/issues/142)
+- Support for configurable chain stop words in ReactAgent
+The ReactAgent YAML configuration now supports the stop_sequence keyword, allowing users to customize chain stop words. For more details, please refer to [issue127](https://github.com/alipay/agentUniverse/issues/127)
+- Added an introduction to RAG principles and a quick guide for building RAG, please pay attention to the corresponding parts in the README and user guide.
+- Added advanced guidance documents for the intelligent agent productization platform, please pay attention to the corresponding parts in the README and user guide.
+- Various code optimizations and documentation updates.
+
 ## [0.0.12] - 2024-08-14
 ### Added
 - agentUniverse Product Version Offering

CHANGELOG_zh.md

@@ -24,6 +24,27 @@ Note - 对于版本的额外说明。
 ***************************************************
 
 # 版本更新记录
+## [0.0.13] - 2024-09-12
+### Added
+- RAG检索增强能力组件版本更新
+- 本版本提供知识库构建、RAG检索召回环节标准作业流程，组件覆盖数据加载、数据处理、索引构建、知识入库、意图改写、检索重排等一系列RAG原子能力，帮助用户在开源场景下快速构建通用RAG智能体方案。
+
+- 智能体产品化平台更新
+- 本版本新增支持智能体画布编排、私有知识库构建、自定义插件等能力，通过低代码、可视化的方式帮助用户快速构建与编排智能体。
+
+- 新增智谱GLM默认模型组件
+- 新增SQLiteStore存储组件
+- 新增flow编排执行引擎
+
+### Note 
+- system_db_uri默认路径优化
+  - 默认路径已兼容windows平台，详情见[issue142](https://github.com/alipay/agentUniverse/issues/142)
+- ReactAgent支持链停止词配置化
+  - ReactAgent yaml配置目前已经支持stop_sequence关键词，用户可以自行配置链停止词,详情见[issue127](https://github.com/alipay/agentUniverse/issues/127)
+- 新增RAG原理介绍与RAG快速构建指导文档，请关注README与用户指南相应部分。
+- 新增智能体产品化平台高阶指导文档，请关注README与用户指南相应部分。
+- 部分代码优化与文档更新
+
 ## [0.0.12] - 2024-08-14
 ### Added
 - agentUniverse产品化版本提供

README.md

@@ -5,7 +5,7 @@ Language version: [English](./README.md) | [中文](./README_zh.md) | [日本語
 ![](https://img.shields.io/badge/framework-agentUniverse-pink)
 ![](https://img.shields.io/badge/python-3.10%2B-blue?logo=Python)
 [![](https://img.shields.io/badge/%20license-Apache--2.0-yellow)](LICENSE)
-[![Static Badge](https://img.shields.io/badge/pypi-v0.0.12-blue?logo=pypi)](https://pypi.org/project/agentUniverse/)
+[![Static Badge](https://img.shields.io/badge/pypi-v0.0.13-blue?logo=pypi)](https://pypi.org/project/agentUniverse/)
 
 ![](docs/guidebook/_picture/logo_bar.jpg)
 ****************************************
@@ -67,15 +67,15 @@ pip install magent-ui ruamel.yaml
 
 Run the [product_application.py](sample_standard_app/app/bootstrap/product_application.py) file located in sample_standard_app/app/bootstrap for a one-click start.
 
-For more details, refer to [Quick Start for Product Platform](./docs/guidebook/en/10_1_1_Product%20Platform%20Quick%20Start.md).
+For more details, refer to [Quick Start for Product Platform](./docs/guidebook/en/10_1_1_Product%20Platform%20Quick%20Start.md) and the [Advanced Guide](./docs/guidebook/en/10_1_2_Product_Platform_Advancement_Guide.md).
 
 This feature is jointly launched by [difizen](https://github.com/difizen/magent) and agentUniverse.
 
 ****************************************
 
 ## Cases and Example Projects
 ### 🌟 Use Cases
-[Legal Consultation Agent](./docs/guidebook/en/7_1_1_Legal_Consultation_Case.md)
+[Legal Consultation Agent v2](./docs/guidebook/en/7_1_1_Legal_Consultation_Case.md)
 
 [Python Code Generation and Execution Agent](./docs/guidebook/en/7_1_1_Python_Auto_Runner.md)
 
@@ -85,6 +85,9 @@ This feature is jointly launched by [difizen](https://github.com/difizen/magent)
 
 [Andrew Ng's Reflexive Workflow Translation Agent Replication](./docs/guidebook/en/7_1_1_Translation_Case.md)
 
+#### 🚩 How to Quickly Build an RAG Agent
+The RAG components have been fully upgraded. This tutorial provides a guide on how to quickly build an RAG agent in agentUniverse. You can refer to the documentation on [How to Build a RAG Agent](./docs/guidebook/en/2_2_4_How_To_Build_A_RAG_Agent.md). For more theoretical content, please check the documentation on [Introduction to RAG](./docs/guidebook/en/2_2_4_RAG.md).
+
 #### 🚩 DataAgent - Data Autonomous Agent
 agentUniverse has launched DataAgent (Minimum Viable Product Version). DataAgent aims to empower your agent with the capability of self-assessment and evolution through the use of intelligent agent abilities. For more details, please refer to the documentation. [DataAgent - Data Autonomous Agent](./docs/guidebook/en/8_1_1_data_autonomous_agent.md)
 

README_jp.md

@@ -5,7 +5,7 @@
 ![](https://img.shields.io/badge/framework-agentUniverse-pink)
 ![](https://img.shields.io/badge/python-3.10%2B-blue?logo=Python)
 [![](https://img.shields.io/badge/%20license-Apache--2.0-yellow)](LICENSE)
-[![Static Badge](https://img.shields.io/badge/pypi-v0.0.10-blue?logo=pypi)](https://pypi.org/project/agentUniverse/)
+[![Static Badge](https://img.shields.io/badge/pypi-v0.0.13-blue?logo=pypi)](https://pypi.org/project/agentUniverse/)
 
 ![](docs/guidebook/_picture/logo_bar.jpg)
 ****************************************

README_zh.md

@@ -5,7 +5,7 @@
 ![](https://img.shields.io/badge/framework-agentUniverse-pink)
 ![](https://img.shields.io/badge/python-3.10%2B-blue?logo=Python)
 [![](https://img.shields.io/badge/%20license-Apache--2.0-yellow)](LICENSE)
-[![Static Badge](https://img.shields.io/badge/pypi-v0.0.12-blue?logo=pypi)](https://pypi.org/project/agentUniverse/)
+[![Static Badge](https://img.shields.io/badge/pypi-v0.0.13-blue?logo=pypi)](https://pypi.org/project/agentUniverse/)
 
 ![](docs/guidebook/_picture/logo_bar.jpg)
 ****************************************
@@ -65,13 +65,14 @@ pip install magent-ui ruamel.yaml
 运行sample_standard_app/app/boostrap下的[product_application.py](sample_standard_app/app/bootstrap/product_application.py)文件，一键启动。
 
 更多详情参考 [产品化平台快速开始](./docs/guidebook/zh/10_1_1_产品化平台快速开始.md)
+与 [产品化平台进阶指南](./10_1_2_产品化平台进阶指南.md) 。
 
 本功能由 [difizen](https://github.com/difizen/magent) X agentUniverse联合推出。
 
 ****************************************
 ## 案例与样例工程
 ### 🌟 使用案例
-[法律咨询Agent](./docs/guidebook/zh/7_1_1_法律咨询案例.md)
+[法律咨询Agent_v2](./docs/guidebook/zh/7_1_1_法律咨询案例.md)
 
 [Python代码生成与执行Agent](./docs/guidebook/zh/7_1_1_Python自动执行案例.md)
 
@@ -81,6 +82,8 @@ pip install magent-ui ruamel.yaml
 
 [吴恩达反思工作流翻译智能体复刻](./docs/guidebook/zh/7_1_1_翻译案例.md)
 
+#### 🚩 如何快速构建RAG智能体
+RAG组件全面升级，教程提供一个在agentUniverse中快速构建一个RAG智能体，可查看文档[如何构建RAG智能体](./docs/guidebook/zh/2_2_4_如何构建RAG智能体.md)，更多原理性内容可查看文档[RAG介绍](./docs/guidebook/zh/2_2_4_RAG.md)。
 
 #### 🚩 DataAgent - 数据自治智能体
 agentUniverse推出了DataAgent（Minimum Viable Product版本）, DataAgent旨在使用智能体能力让您的Agent拥有自我评价与演进的能力。详情见文档: [DataAgent - 数据自治智能体](./docs/guidebook/zh/8_1_1_数据自治智能体.md)

agentuniverse/agent/action/knowledge/doc_processor/__init__.py

@@ -0,0 +1,7 @@
+# !/usr/bin/env python3
+# -*- coding:utf-8 -*-
+
+# @Time    : 2024/7/23 13:59
+# @Author  : fanen.lhy
+# @Email   : fanen.lhy@antgroup.com
+# @FileName: __init__.py.py

agentuniverse/agent/action/knowledge/doc_processor/character_text_splitter.py

@@ -0,0 +1,50 @@
+# !/usr/bin/env python3
+# -*- coding:utf-8 -*-
+
+# @Time    : 2024/8/5 14:37
+# @Author  : fanen.lhy
+# @Email   : fanen.lhy@antgroup.com
+# @FileName: text_splitter.py
+from typing import List, Optional
+from langchain.text_splitter import CharacterTextSplitter as Splitter
+
+from agentuniverse.agent.action.knowledge.doc_processor.doc_processor import \
+    DocProcessor
+from agentuniverse.agent.action.knowledge.store.document import Document
+from agentuniverse.agent.action.knowledge.store.query import Query
+from agentuniverse.base.config.component_configer.component_configer import \
+    ComponentConfiger
+
+
+class CharacterTextSplitter(DocProcessor):
+    chunk_size: int = 200
+    chunk_overlap: int = 20
+    separator: str = "/n/n"
+    splitter: Optional[Splitter] = None
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self.splitter = Splitter(separator=self.separator,
+                                 chunk_size=self.chunk_size,
+                                 chunk_overlap=self.chunk_overlap)
+
+    def _process_docs(self, origin_docs: List[Document], query: Query = None) -> \
+            List[Document]:
+        lc_doc_list = self.splitter.split_documents(Document.as_langchain_list(
+            origin_docs
+        ))
+        return Document.from_langchain_list(lc_doc_list)
+
+    def _initialize_by_component_configer(self,
+                                         doc_processor_configer: ComponentConfiger) -> 'DocProcessor':
+        super()._initialize_by_component_configer(doc_processor_configer)
+        if hasattr(doc_processor_configer, "chunk_size"):
+            self.chunk_size = doc_processor_configer.chunk_size
+        if hasattr(doc_processor_configer, "chunk_overlap"):
+            self.chunk_overlap = doc_processor_configer.chunk_overlap
+        if hasattr(doc_processor_configer, "separator"):
+            self.separator = doc_processor_configer.separator
+        self.splitter = Splitter(separator=self.separator,
+                                 chunk_size=self.chunk_size,
+                                 chunk_overlap=self.chunk_overlap)
+        return self

agentuniverse/agent/action/knowledge/doc_processor/character_text_splitter.yaml

@@ -0,0 +1,9 @@
+name: 'character_text_splitter'
+description: 'langchain character text splitter'
+chunk_size: 200
+chunk_overlap: 20
+separators: "/n/n"
+metadata:
+  type: 'DOC_PROCESSOR'
+  module: 'agentuniverse.agent.action.knowledge.doc_processor.character_text_splitter'
+  class: 'CharacterTextSplitter'

agentuniverse/agent/action/knowledge/doc_processor/dashscope_reranker.py

@@ -0,0 +1,67 @@
+# !/usr/bin/env python3
+# -*- coding:utf-8 -*-
+
+# @Time    : 2024/8/5 15:48
+# @Author  : fanen.lhy
+# @Email   : fanen.lhy@antgroup.com
+# @FileName: dashscope_reranker.py
+
+from typing import List, Optional
+import dashscope
+from http import HTTPStatus
+
+from agentuniverse.agent.action.knowledge.doc_processor.doc_processor import \
+    DocProcessor
+from agentuniverse.agent.action.knowledge.store.document import Document
+from agentuniverse.agent.action.knowledge.store.query import Query
+from agentuniverse.base.config.component_configer.component_configer import \
+    ComponentConfiger
+
+MODEL_NAME_MAP = {
+    "gte_rerank": dashscope.TextReRank.Models.gte_rerank
+}
+
+
+class DashscopeReranker(DocProcessor):
+    model_name: str = "gte_rerank"
+    top_n: int = 10
+
+    def _process_docs(self, origin_docs: List[Document], query: Query = None) -> \
+            List[Document]:
+        if not query or not query.query_str:
+            raise Exception("Dashscope reranker need an origin string query.")
+        if len(origin_docs)<1:
+            return origin_docs
+        documents_texts = []
+        for _doc in origin_docs:
+            documents_texts.append(_doc.text)
+        resp = dashscope.TextReRank.call(
+            model=MODEL_NAME_MAP.get(self.model_name),
+            query=query.query_str,
+            documents=documents_texts,
+            top_n=self.top_n,
+            return_documents=False
+        )
+        if resp.status_code == HTTPStatus.OK:
+            results = resp.output.results
+        else:
+            raise Exception(f"Dashscope rerank api call error: {resp}")
+        rerank_docs = []
+        for _result in results:
+            index = _result.index
+            if origin_docs[index].metadata:
+                origin_docs[index].metadata["relevance_score"] = _result.relevance_score
+            else:
+                origin_docs[index].metadata = {"relevance_score": _result.relevance_score}
+            rerank_docs.append(origin_docs[index])
+
+        return rerank_docs
+
+    def _initialize_by_component_configer(self,
+                                         doc_processor_configer: ComponentConfiger) -> 'DocProcessor':
+        super()._initialize_by_component_configer(doc_processor_configer)
+        if hasattr(doc_processor_configer, "model_name"):
+            self.model_name = doc_processor_configer.model_name
+        if hasattr(doc_processor_configer, "top_n"):
+            self.top_n = doc_processor_configer.top_n
+        return self

agentuniverse/agent/action/knowledge/doc_processor/dashscope_reranker.yaml

@@ -0,0 +1,6 @@
+name: 'dashscope_reranker'
+description: 'reranker use dashscope api'
+metadata:
+  type: 'DOC_PROCESSOR'
+  module: 'agentuniverse.agent.action.knowledge.doc_processor.dashscope_reranker'
+  class: 'DashscopeReranker'

agentuniverse/agent/action/knowledge/doc_processor/doc_processor.py

@@ -0,0 +1,56 @@
+# !/usr/bin/env python3
+# -*- coding:utf-8 -*-
+
+# @Time    : 2024/7/23 14:00
+# @Author  : fanen.lhy
+# @Email   : fanen.lhy@antgroup.com
+# @FileName: doc_processor.py
+
+from abc import abstractmethod
+from typing import List, Optional
+
+from agentuniverse.agent.action.knowledge.store.query import Query
+from agentuniverse.agent.action.knowledge.store.document import Document
+from agentuniverse.base.component.component_base import ComponentEnum
+from agentuniverse.base.component.component_base import ComponentBase
+from agentuniverse.base.config.component_configer.component_configer import \
+    ComponentConfiger
+
+
+class DocProcessor(ComponentBase):
+    """The basic class for doc processor.
+    """
+
+    component_type: ComponentEnum = ComponentEnum.DOC_PROCESSOR
+    name: Optional[str] = None
+    description: Optional[str] = None
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    def process_docs(self, origin_docs: List[Document], query: Query = None) -> \
+            List[Document]:
+        """Process input documents，return should also be a document list."""
+        return self._process_docs(origin_docs, query)
+
+    @abstractmethod
+    def _process_docs(self, origin_docs: List[Document],
+                      query: Query = None) -> \
+            List[Document]:
+        """Process input documents，return should also be a document list."""
+        pass
+
+    def _initialize_by_component_configer(self,
+                                         doc_processor_configer: ComponentConfiger) \
+            -> 'DocProcessor':
+        """Initialize the DocProcessor by the ComponentConfiger object.
+
+        Args:
+            doc_processor_configer(ComponentConfiger): A configer contains DocProcessor
+            basic info.
+        Returns:
+            DocProcessor: A DocProcessor instance.
+        """
+        self.name = doc_processor_configer.name
+        self.description = doc_processor_configer.description
+        return self

agentuniverse/agent/action/knowledge/doc_processor/doc_processor_manager.py

@@ -0,0 +1,20 @@
+# !/usr/bin/env python3
+# -*- coding:utf-8 -*-
+
+# @Time    : 2024/7/23 14:12
+# @Author  : fanen.lhy
+# @Email   : fanen.lhy@antgroup.com
+# @FileName: doc_processor_manager.py
+
+from agentuniverse.base.annotation.singleton import singleton
+from agentuniverse.base.component.component_enum import ComponentEnum
+from agentuniverse.base.component.component_manager_base import ComponentManagerBase
+from agentuniverse.agent.action.knowledge.doc_processor.doc_processor import DocProcessor
+
+
+@singleton
+class DocProcessorManager(ComponentManagerBase[DocProcessor]):
+    """A singleton manager class of the DocProcessor."""
+
+    def __init__(self):
+        super().__init__(ComponentEnum.DOC_PROCESSOR)