FederatedAI · Redias · Sep 8, 2023 · Sep 8, 2023 · Sep 8, 2023 · Sep 8, 2023
diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml
@@ -5,6 +5,7 @@ on:
     branches:
       - 'main'
       - 'develop-[0-9]+.[0-9]+.[0-9]+'
+      - 'build-doc-[0-9]+.[0-9]+.[0-9]+-[a-zA-Z]+'
 
   schedule:
     - cron: '0 8 * * *'
@@ -41,6 +42,7 @@ jobs:
           VERSION='${{ github.ref_name }}'
           [ "$VERSION" == main ] && { VERSION=latest; ALIAS='main master'; }
           VERSION="${VERSION#develop-}"
+          VERSION="${VERSION#build-doc-}"
 
           mike deploy --push --update-aliases "$VERSION" $ALIAS
           mike set-default --push latest
diff --git a/.gitignore b/.gitignore
@@ -13,15 +13,15 @@ venv
 
 # excluded paths
 /data/
+/model/
 /logs/
 /jobs/
 /audit/
+/localfs/
 .vscode/*
 /temp/
 /tmp
 /worker/
-/provider_registrar/
-/model_local_cache/
 *.db
 *.db-journal
 *.whl
@@ -31,3 +31,14 @@ venv
 
 # doc
 /site/
+
+/python/fate_flow/data
+/python/fate_flow/model
+/python/fate_flow/logs
+/python/fate_flow/jobs
+/python/fate_flow/localfs
+/python/fate_flow/*.env
+/python/fate_flow/conf
+/python/build
+/python/dist
+/python/*.egg-info
diff --git a/README.md b/README.md
@@ -20,8 +20,6 @@ Providing production-level service capabilities:
 - High Availability
 - CLI, REST API, Python API
 
-For detailed introduction, please refer to [FATE Flow Overall Design](https://federatedai.github.io/FATE-Flow/latest/fate_flow/#overall-design)
-
 ## Deployment
 
 Please refer to [FATE](https://github.com/FederatedAI/FATE)

diff --git a/README.zh.md b/README.zh.md
@@ -4,7 +4,7 @@
 
 FATE Flow是一个联邦学习端到端全流程的多方联合任务安全调度平台, 基于:
 
-- [共享状态调度架构](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/41684.pdf)
+- 共享状态调度架构
 - 跨数据中心的多方安全通信
 
 提供生产级服务能力:
@@ -20,7 +20,6 @@ FATE Flow是一个联邦学习端到端全流程的多方联合任务安全调
 - 系统高可用
 - CLI、REST API、Python API
 
-详细介绍请参考[FATE Flow整体设计](https://federatedai.github.io/FATE-Flow/latest/zh/fate_flow/)
 
 ## 部署
 

diff --git a/RELEASE.md b/RELEASE.md
@@ -1,259 +1,29 @@
-# Release 1.11.2
-## Major Features and Improvements
-* Support real-time log retrieval and display for FATE LLM tasks.
-* Optimize the logic of the job clean interface.
-
-## Bug Fixes
-* Fix the thread accumulation caused by the session cleanup timeout bug.
-
-# Release 1.11.1
-## Major Features and Improvements
-* Support distributed training with multiple gpus for FATE-LLM by Eggroll
-
-## Bug Fixes
-* Fix hadoop connection failures in some scenarios
-* Fix spark config in role does not take effect
-
-# Release 1.11.0
-## Major Features and Improvements
-* Add data table preview query interface
-
-## Bug Fixes
-* Fix the performance problems of upload and reader in processing large amounts of data
-* Fix online inference cannot be done after model migration bug
-* Fix the model cannot be saved to the specified database bug
-* Fix reader data preview display bug
-
-
-# Release 1.10.1
-## Major Features and Improvements
-* Optimize table info API
-
-
-# Release 1.10.0
-## Major Features and Improvements
-* Add connection test API
-* May configure gRPC message size limit 
-## Bug Fixes
-* Fix module duplication issue in model 
-
-# Release 1.9.1
-## Bug Fixes
-* Fix parameter inheritance when loading non-model modules from ModelLoader 
-* Fix job inheritance after adding or removing roles from training configuration
-* Fix delimiter error in uploaded/downloaded data
-* Fix anonymous feature name renewal
-
-# Release 1.9.0
-## Major Features and Improvements
-* Support high availability and load balancing to improve system availability and stability
-* Added support for site authentication and data set authority authentication, and supports hook mode for users to customize authentication schemes
-* Component registration optimization, support participants to use different versions of algorithm components
-* Upload, reader support feature anonymity, support specifying id column
-* Scheduling optimization, asynchronous time-consuming operations, component scheduling performance improved by more than 5 times This optimization obvious benefits for multi-component tasks
-* Added component ApiReader to get feature data by id
-* Model storage optimization, support model data synchronization between local and other storage
-* The scheduler now can obtain the error information from other participant's algorithm components
-
-# Release 1.8.0
-## Major Features and Improvements
-* Optimize the model migration function to reduce user operation steps;
-* Add version compatibility check in component center to support multiple parties to use different versions;
-* Add data table disable/enable function, and support batch delete disable table
-
-# Release 1.7.2
-## Major Features and Improvements
-* Separate the base connection address of the data storage table from the data table information, and compatible with historical versions;
-* Optimize the component output data download interface.
-
-# Release 1.7.1
-## Major Features and Improvements
-* Added the writer component, which supports exporting data to mysql and saving data as a new table;
-* Added job reuse function, which supports the reuse of successful status components of historical tasks in new jobs;
-* Optimize the time-consuming problem of submitting tasks and the time-consuming problem of stopping tasks;
-* Component registration supports automatic setting of PYTHONPYTH.
-
-## Bug Fixes
-* Fix the problem of OOM when uploading hdfs table;
-* Fix the problem of incompatibility with the old version of serving;
-* The parameter partitions of the toy test is set to 4, and a timeout prompt is added.
-
-# Release 1.7.0
-
-## Major Features and Improvements
-
-* Independent repository instead of all code in the main FATE repository
-* Component registry, which can hot load many different versions of component packages at the same time
-* Hot update of component parameters, component-specific reruns, automatic reruns
-* Model Checkpoint to support task hot start, model deployment and other
-* Data, Model and Cache can be reused between jobs
-* Reader component supports more data sources, such as MySQL, Hive
-* Realtime recording of dataset usage derivation routes
-* Multi-party permission control for datasets
-* Automatic push to reliable storage when model deployment, support Tencent Cloud COS, MySQL, Redis
-* REST API authentication
-
-## Bug Fixes
-
-# Release 1.6.1
-## Major Features and Improvements
-* Support mysql storage engine;
-* Added service registry interface;
-* Added service query interface;
-* Support fate on WeDataSphere mode
-* Add lock when writing `model_local_cache`
-* Register the model download urls to zookeeper
-
-## Bug Fixes
-* Fix job id length no more than 25 limitation
-
-
-# Release 1.5.2
-## Major Features and Improvements
-* Read data from mysql with ‘table bind’ command to map source table to FATE table
-* FATE cluster push model for one-to-multiple FATE Serving clusters in one party
-
-## Bug Fixes
-* Fix job id length no more than 25 limitation
-
-
-# Release 1.5.1
-## Major Features and Improvements
-* Optimize the model center, reconstruct publishing model, support deploy, load, bind, migrate operations, and add new interfaces such as model info
-* Improve identity authentication and resource authorization, support party identity verification, and participate in the authorization of roles and components
-* Optimize and fix resource manager, add task_cores job parameters to adapt to different computing engines
-
-## Deploy
-* Support 1.5.0 retain data upgrade to 1.5.1
-
-## Bug Fixes
-* Fix job clean CLI
-
-
-# Release 1.5.0（LTS）
-## Major Features and Improvements
-* Brand new scheduling framework based on global state and optimistic concurrency control and support multiple scheduler
-* Upgraded task scheduling: multi-model output for component, executing component in parallel, component rerun
-* Add new DSL v2 which significantly improves user experiences in comparison to DSL v1. Several syntax error detection functions are supported in v2. Now DSL v1 and v2 are 
-   compatible in the current FATE version
-* Enhanced resource scheduling: remove limit on job number, base on cores, memory and working node according to different computing engine supports
-* Add model registry, supports model query, import/export, model transfer between clusters
-* Add Reader component: automatically dump input data to FATE-compatible format and cluster storage engine; now data from HDFS
-* Refactor submit job configuration's parameters setting, support different parties use different job parameters when using dsl V2.
-
-## Client
-* Brand new CLI v2 with easy independent installation, user-friendly programming syntax & command-line prompt
-* Support FLOW python language SDK
-
-
-# Release 1.4.4
-## Major Features and Improvements
-* Task Executor supports monkey patch
-* Add forward API
-
-
-# Release 1.4.2
-## Major Features and Improvements
-* Distinguish between user stop job and system stop job;
-* Optimized some logs;
-* Optimize zookeeper configuration
-* The model supports persistent storage to mysql
-* Push the model to the online service to support the specified storage address (local file and FATEFlowServer interface)
-
-
-# Release 1.4.1
-## Major Features and Improvements
-* Allow the host to stop the job
-* Optimize the task queue
-* Automatically align the input table partitions of all participants when the job is running
-* Fate flow client large file upload optimization
-* Fixed some bugs with abnormal status
-
-
-# Release 1.4.0
-## Major Features and Improvements
-* Refactoring model management, native file directory storage, storage structure is more flexible, more information
-* Support model import and export, store and restore with reliable distributed system(Redis is currently supported)
-* Using MySQL instead of Redis to implement Job Queue, reducing system complexity
-* Support for uploading client local files
-* Automatically detects the existence of the table and provides the destroy option
-* Separate system, algorithm, scheduling command log, scheduling command log can be independently audited
-
-
-# Release 1.3.1
-## Major Features and Improvements
-## Deploy
-* Support deploying by MacOS
-* Support using external db
-* Deploy JDK and Python environments on demand
-* Improve MySQL and FATE Flow service.sh
-* Support more custom deployment configurations in the default_configurations.sh, such as ssh_port, mysql_port and so one.
-
-# Release 1.3.0
-## Major Features and Improvements
-* Add clean job CLI for cleaning output and intermediate results, including data, metrics and sessions
-* Support for obtaining table namespace and name of output data via CLI
-* Fix KillJob unsuccessful execution in some special cases
-* Improve log system, add more exception and run time status prompts
-
-
-# Release 1.2.0
-## Major Features and Improvements
-* Add data management module for recording the uploaded data tables and the outputs of the model in the job running, and for querying and cleaning up CLI. 
-* Support registration center for simplifying communication configuration between FATEFlow and FATEServing
-* Restruct model release logic, FATE_Flow pushes model directly to FATE-Serving. Decouple FATE-Serving and Eggroll, and the offline and online architectures are connected only by FATE-Flow.
-* Provide CLI to query data upload record
-* Upload and download data support progress statistics by line
-* Add some abnormal diagnosis tips
-* Support adding note information to job
-
-## Deploy
-* Fix bugs in EggRoll startup script, add mysql, redis startup options.
-* Disable host name resolution configuration for mysql service.
-* The version number of each module of the software packaging script is updated using the automatic acquisition mode.
-
-
-# Release 1.1.1
-## Major Features and Improvements
-* Add cluster deployment support based on ubuntu operating system。
-* Support intermediate data cleanup after the task ends
-* Optimizing the deployment process
-
-
-## Bug Fixes
-* Fix a bug in download api
-* Fix bugs of spark-backend
-
-
-# Release 1.1
-## Major Features and Improvements
-* Upload and Download support CLI for querying job status
-* Support for canceling waiting job
-* Support for setting job timeout
-* Support for storing a job scheduling log in the job log folder
-* Add authentication control Beta version, including component, command, role
-
-
-# Release 1.0.2
-## Major Features and Improvements
-* Python and JDK environment are required only for running standalone version quick experiment
-* Support cluster version docker deployment
-* Add deployment guide in Chinese
-* Standalone version job for quick experiment is supported when cluster version deployed. 
-* Python service log will remain for 14 days now.
-
-
-# Release 1.0.1
-## Bug Fixes
-* Support upload file in version argument
-* Support get serviceRoleName from configuration
-
-
-# Release 1.0
-## Major Features and Improvements
-* DAG defines Pipeline
-* Federated Multi-party asymmetric DSL parser
-* Federated Learning lifecycle management
-* Federated Task collaborative scheduling
-* Tracking for data, metric, model and so on
-* Federated Multi-party model management
+## Release 2.2.0
+### Major Features and Improvements
+* Upgrade from Python 3.8 to Python 3.10
+
+## Release 2.1.0
+### Major Features and Improvements
+Improved the display issue of output data.
+Enhanced the PyPI package: configuration files have been relocated to the user's home directory, and the relative paths for uploading data are based on the user's home directory.
+Supported running FATE algorithms in Spark on YARN client mode.
+
+### Bug-Fix
+Fixed an issue where failed tasks could not be retried.
+Fixed an issue where the system couldn't run when the task cores exceeded the system total cores.
+
+## Release 2.0.0
+### Major Features and Improvements
+* Adapted to new scalable and standardized federated DSL IR
+* Built an interconnected scheduling layer framework, supported the BFIA protocol
+* Optimized process scheduling, with scheduling separated and customizable, and added priority scheduling
+* Optimized algorithm component scheduling，support container-level algorithm loading， enhancing support for cross-platform heterogeneous scenarios
+* Optimized multi-version algorithm component registration, supporting registration for mode of components
+* Federated DSL IR extension enhancement: supports multi-party asymmetric scheduling
+* Optimized client authentication logic, supporting permission management for multiple clients
+* Optimized RESTful interface, making parameter fields and types, return fields, and status codes clearer
+* Added OFX(Open Flow Exchange) module: encapsulated scheduling client to allow cross-platform scheduling
+* Supported the new communication engine OSX, while remaining compatible with all engines from FATE Flow 1.x
+* Decoupled the System Layer and the Algorithm Layer, with system configuration moved from the FATE repository to the Flow repository
+* Published FATE Flow package to PyPI and added service-level CLI for service management
+* Migrated major functionality from FATE Flow 1.x
diff --git a/bin/init_env.sh b/bin/init_env.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+#
+#  Copyright 2019 The FATE Authors. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+fate_project_base=$(cd `dirname "$(realpath "${BASH_SOURCE[0]:-${(%):-%x}}")"`; cd ../;cd ../;pwd)
+export FATE_PROJECT_BASE=$fate_project_base
+export PYTHONPATH=
+export SPARK_HOME=
+venv=
+
+source ${venv}/bin/activate
+