add cbdb overview

docs/cbdb-overview-new.md

@@ -0,0 +1,110 @@
+---
+id: cbdb-overview
+title: Product Introduction
+---
+
+# Cloudberry Database Introduction
+
+Cloudberry Database, built on the latest PostgreSQL 14.4 core (released in mid-2022), stands as one of the most advanced and mature open-source MPP databases available. It encompasses a range of high-performance features, including high concurrency and high availability, to efficiently handle complex tasks and meet the demands of managing and processing massive amounts of data. Its widespread adoption across various domains speaks to its extensive applicability.
+
+- Outstanding performance: Cloudberry exhibits significant advantages in data storage, high concurrency, high availability, linear scalability, responsiveness, ease of use, and cost-effectiveness. In the era of big data, Cloudberry excels in processing terabyte-level datasets, surpassing Hadoop in terms of stand-alone performance.
+- Strong syntax compatibility: Cloudberry offers superior usability and functionality compared to Hive, the SQL engine on Hadoop. Its user-friendly approach makes it more accessible for both novice and experienced users.
+- Comprehensive tooling: Cloudberry boasts a comprehensive suite of tools, eliminating the need for extensive tool customization. This makes it an ideal solution for large-scale data warehouse projects, saving time and effort for users.
+- Flexible deployment: Cloudberry supports flexible deployment options, including the traditional hardware deployment, as well as the multi-cloud and cross-cloud deployments.
+- Cloudberry provides comprehensive support for diverse data types, formats, and storage media. Its flexibility ensures it can effectively meet the various requirements of users.
+
+This manual focuses on introducing the product architecture and the underlying mechanisms of Cloudberry Database's internal modules, and highlights their significance for users.
+
+## Product architecture
+
+In most cases, Cloudberry Database is similar to PostgreSQL in terms of SQL support, functionality, configuration options, and end-user capabilities. The interactive experience of users with Cloudberry Database is very close to the interaction with stand-alone PostgreSQL.
+
+Cloudberry Database uses MPP (Massively Parallel Processing) architecture technology to store and process vast amounts of data by distributing data and workloads across multiple servers or hosts.
+
+MPP, also known as a shared-nothing architecture, refers to a system consisting of multiple hosts that collaborate to execute operations. Each host has its own processor, memory, disk, network resources, and operating system. Cloudberry Database leverages this high-performance system architecture to efficiently distribute the workload of massive data and fully utilize all system resources in parallel for query processing.
+
+From a user's perspective, Cloudberry Database is a comprehensive Relational Database Management System (RDBMS). At the physical level, it contains multiple PostgreSQL instances. To achieve efficient division of labor and cooperation among multiple independent PostgreSQL instances, Cloudberry Database implements distributed clustering in data storage, computation, communication, and management. Cloudberry Database encapsulates all the intricate distributed details, providing users with a unified logical database. This encapsulation significantly simplifies the work of developers and operations personnel.
+
+The architecture diagram of Cloudberry Database is shown below: [Insert Architecture Diagram]
+
+![Cloudberry architecture](./media/cbdb-arch.png)
+
+HashData is comprised of the following components:
+
+- The **master** serves as the gateway to Cloudberry Database. It handles client connections and SQL queries, and distributes tasks to segment instances. Users interact with Cloudberry Database by connecting to the master using client programs (such as psql) or application programming interfaces (APIs) like JDBC, ODBC, or the libpq PostgreSQL C API.
+    - The master acts as the global system directory, containing a set of system tables that hold metadata of Cloudberry Database. 
+    - The master doesn't store any user data. User data is exclusively stored on segment instances.
+    - The master performs client connection authentication, processes SQL commands, distributes workload among segments, coordinates the results returned by each segment, and presents the final results to the client program.
+    - Cloudberry Database utilizes Write-Ahead Logging (WAL) for master/standby mirroring. In WAL-based logging, all modifications are logged before they are written to disk to ensure data integrity for any in-process operations.
+
+- The **segment** instances are independent Postgres processes. Each segment stores a specific portion of the data and handles the corresponding queries. When a user connects to the database through the master and submits a query request, processes are created on each segment to handle the query. User-defined tables and their indexes are distributed across the available segments, and each data node contains distinct portions of the data. The processes of data processing runs in the corresponding segment. Users interact with segments through the master, and the segment operate on servers known as the segment host.
+
+    Typically, a segment host runs 2 to 8 segment instances, depending on the processor, memory, storage, network interfaces and workload. Balancing the configuration of segment hosts is critical because Cloudberry Database achieves optimal performance by evenly distributing data and workload among segments, allowing all segments to initiate and complete tasks simultaneously.
+
+- The **interconnection** serves as the network layer in the Cloudberry Database system architecture. It refers to the network infrastructure on which the communication of the master and segments depends, using a standard Ethernet switching structure.
+
+    For performance reasons, a 10 GB or faster network is recommended. By default, the interconnection module communicates using the UDP protocol with flow control (UDPIFC) to send messages over the network. Cloudberry Database performs packet validation beyond what UDP offers, which means reliability of Cloudberry equivalent to using the TCP protocol, and performance and scalability exceeds that of TCP. If the interconnection is changed to the TCP protocol instead, Cloudberry Database's scalability is limited to 1000 segments. However, this restriction does not apply when UDPIFC is used as the default protocol.
+
+- Cloudberry Database uses Multiversion Concurrency Control (MVCC) to ensure data consistency. When querying the database, each transaction sees only a snapshot of the data, which ensures that the current transaction does not see other transactions modifying the same records. Accordingly, MVCC provides transaction isolation for every database transaction.
+
+    MVCC minimizes lock contention to ensure performance in multi-user environments by avoiding explicit locking for database transactions. One significant advantage of MVCC over locks is that read and write operations do not conflict, and they never block each other.
+
+## Data loading
+
+Cloudberry Database supports massive parallel and persistent data loading through external table technology, and enables automatic conversion between character sets such as GBK and UTF8. Based on the MPP architecture, the Scatter-Gather Streaming<sup>TM</sup> technology provides linear expansion of performance. CBDB supports a variety of storage media such as external file servers, Hive, Hbase, HDFS, S3 and various file formats such as CSV, Text, JSON, ORC, Parquet, and supports compressed data file loading such as Zip. CBDB is used by DataStage, Informatica, Kettle for ETL tool integration.
+
+Cloudberry Database also supports streaming data loading. For subscribed Kafka Topics, it launches multiple tasks in parallel to read partition data based on the configured maximum task value. After reading the data, the records are cached until a certain time or record count is reached. They are then loaded into Cloudberry Database using gpfdist to ensure data integrity and prevent duplication or loss. This capability is ideal for scenarios involving streaming data collection and real-time analysis. Cloudberry Database supports a data loading throughput of tens of millions of records per minute.
+
+PXF is a built-in component of Cloudberry Database, which supports mapping external data sources to Cloudberry Database external tables to implement the Data Fabric architecture. Parallel and high-speed data access is realized based on the MPP engine, and hybrid data ecological management and access are supported.
+
+![Data loading architecture diagram](./media/cbdb-data-loading-arch.png)
+
+## Data storage and security
+
+Cloudberry Database computation parallelization is based on the even distribution of data in storage layer, and even data distribution is the key to parallel processing. Cloudberry Database provides two types of data distribution methods, Hash and Random, to ensure that data is evenly distributed on each disk, so that each disk's performance can be exploited and I/O bottleneck can be fundamentally solved. Cloudberry Database provides more flexible distribution methods.
+
+Replication Table can be used for small tables, and users can specify custom Hash algorithms when creating tables to control data distribution flexibly. It supports both row storage and column storage.
+
+- Row storage: fast update speed, frequent query for most fields, more random row access.
+- Columnar storage: Few field queries, significant savings in I/O operations, and frequent access to large data volumes.
+
+Cloudberry can design the storage mode according to the application type, down to the finest granularity to partition, to achieve a table with multiple storage modes to optimize access performance. When a query is executed, the Cloudberry Database optimizer generates the corresponding optimal query plan based on statistical information and the storage form used by the user, without user intervention.
+
+![Row storage and column storage](./media/cbdb-row-col-storage.png)
+
+Data compression can improve data processing performance. The compression ratio depends on the compression algorithm and the data content, and can reach more than 20 times for mobile communication signaling, call detail records, and clickstream data. Compression is supported in all storage modes, allowing different columns within a table to utilize distinct compression algorithms. Cloudberry Database provides several compression algorithms:
+
+- Zlib 1-9: high compression ratio, occupies more CPU resources, suitable for scenarios with strong CPU computing power.
+- Zstandard 1~19: achieve the balance between CPU and compression ratio.
+
+Data security is also very important. Cloudberry supports multiple databases, and data is not shared between databases. Cross-database access can be performed through DBLink. The logical organization of data within the database includes multiple types of data objects such as tables, views, indexes, and functions. Data access is supported across schemas.
+
+In terms of storage security, Cloudberry supports different storage modes, data redundancy and data encryption (AES 128, AES 192, AES 256, DES and national secret encryption). Cloudberry supports ciphertext authentication and various encryption algorithms such as SCRAM-SHA-256, MD5, LDAP, RADIUS. For different users, set various types of permissions on different levels of objects (such as schema, table, row, column, view and function). The permissions that can be set include select, update, execution, ownership and more.
+
+## Data analysis
+
+The Cloudberry Database core has a powerful built-in parallel optimizer and executor, which is compatible with the PostgreSQL ecosystem and supports technologies such as data partition pruning, index(such as BTree, Bitmap, Hash, Brin, GIN), and JIT (expression just-in-time compilation processing).
+
+In addition, Cloudberry Database integrates a large number of rich analysis components:
+
+- Machine learning component MADlib on Cloudberry Database: SQL-driven, algorithm + computing power + data.
+- PL language. Developers can write user-defined functions in R, Python, Perl, Java, PostgreSQL, and other languages.
+- Based on the MPP engine, CBDB realizes high-performance, parallel computing, seamlessly integrates with SQL, and executes calculation and analysis on SQL results.
+- PostGIS, based on PostGIS 2.X with enterprise-level improvements, supports Cloudberry Database MPP architecture, integrates object storage, supports large-capacity objects, supports all spatial data types (such as geometry, geography, Raster), supports spatio-temporal index, and supports complex spatial and geographic location calculations, sphere length calculations, and spatial aggregation functions (such as contain, cover, intersect).
+- The Cloudberry Database Text component supports accelerated document retrieval capabilities via ElasticSearch, which has significantly improved the performance of traditional GIN data text query by an order of magnitude, and supports multiple word segmentation, natural language processing, and query result rendering capabilities.
+
+## Flexible workload management
+
+- Connection pool PGBouncer (Connection level, supports high concurrency of Cloudberry clusters at the connection level): the database side manages sessions in a unified manner, controls how many users can access at the same time, avoids frequent creation and destruction of service processes. PGBouncer occupies small memory, supports high concurrency, and uses libevent for socket communication to achieve higher efficiency.
+- Resource group (Session level, quantitative control of Cloudberry cluster resources at the session level): Sorts out workloads, analyzes the CPU and memory of the load, and publishes requirements, sets Resource Group based on workload analysis, monitors GP operation, dynamically adjusts RS, and uses rules to clean up idle sessions.
+- Dynamically allocating resource groups (Query level, dynamically adjust CBDB cluster resources at the SQL level): before or during the execution of SQL statements, dynamically implement flexible and dynamic allocation of resources, give priority to specific queries to shorten their running time.
+
+## Highly compatible with third-party products
+
+Cloudberry Database has excellent connectivity to BI tools, mining and forecasting tools, ETL tools, J2EEa and .NET applications, data sources and computing engines.
+
+![Third party compatible tools](./media/cbdb-third-party-compati.png)
+
+## Cross-platform and localization support
+
+Cloudberry Database supports multiple hardware architectures, including X86, ARM, Feiteng, Kunpeng, and Haikou systems, as well as operating system environments such as CentOS, Ubuntu, Kylin, and BC-Linux.

i18n/zh/docusaurus-plugin-content-docs/current/cbdb-overview.md

@@ -37,11 +37,11 @@ HashData 由如下组件构成：
     - 控制节点对客户端连接进行身份验证，处理传入的 SQL 命令，在数据节点之间分配工作负载，协调每个数据节点返回的结果，并将最终结果呈现给客户端程序。
     - Cloudberry Database 使用预写日志记录（WAL）进行控制节点/Standby 镜像。在基于 WAL 的日志记录中，所有修改都将在写入磁盘之前先写日志，以确保任何进程内操作的数据完整性。
 
-- **数据节点 (Segement)** 实例是独立的 Postgres 进程，每个数据节点存储一部分数据并执行相应部分查询。当用户通过控制节点连接到数据库并提交查询请求时，会在每个数据节点创建进程来处理查询。用户定义的表及其索引分布在 Cloudberry Database 中的所有可用数据节点中，每个数据节点都包含数据的不同部分，不同部分数据处理的进程在相应的数据节点中运行。用户通过控制节点与数据节点进行交互，数据节点在称为数据节点主机的服务器上运行。
+- **数据节点 (Segment)** 实例是独立的 Postgres 进程，每个数据节点存储一部分数据并执行相应部分查询。当用户通过控制节点连接到数据库并提交查询请求时，会在每个数据节点创建进程来处理查询。用户定义的表及其索引分布在 Cloudberry Database 中的所有可用数据节点中，每个数据节点都包含数据的不同部分，不同部分数据处理的进程在相应的数据节点中运行。用户通过控制节点与数据节点进行交互，数据节点在称为数据节点主机的服务器上运行。
 
     数据节点主机通常执行 2 到 8 个数据节点，具体取决于处理器、内存、存储、网络接口和工作负载。数据节点主机的需要平衡配置，因为 Cloudberry Database获得最佳性能的关键是将数据和工作负载平均分配到数据节点中，以便所有数据节点同时开始处理一项任务并同时完成工作。
 
-- **内部互联 (Interconnect)** 是 Cloudberry Database 系统架构中的网络层。内部互联是指控制节点、数据节点通信所依赖的网络基础架构，使用标准的以太网交换结构。
+- **内部互联 (Interconnection)** 是 Cloudberry Database 系统架构中的网络层。内部互联是指控制节点、数据节点通信所依赖的网络基础架构，使用标准的以太网交换结构。
 
     出于性能原因，建议使用 10 GB 或更快的网络。默认情况下，内部互联模块使用带有流控制(UDPIFC) 的 UDP 协议来实现通信，以通过网络发送消息。Cloudberry Database 执行的数据包验证超出了 UDP 所提供的范围，这意味着可靠性等同于使用 TCP 协议，并且性能和可伸缩性超过了 TCP 协议。 如果将内部互联改为使用 TCP 协议，则 Cloudberry Database 的可伸缩性限制为 1000 个数据节点。使用 UDPIFC 作为默认协议时，此限制不适用。