Skip to content

Commit

Permalink
merge(main) merge from main
Browse files Browse the repository at this point in the history
  • Loading branch information
@ni-ze
ni-ze committed Jun 28, 2022
2 parents 2c1d938 + 8ba8e00 commit e5fd612
Show file tree
Hide file tree
Showing 38 changed files with 291 additions and 498 deletions.
86 changes: 50 additions & 36 deletions README-chinese.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
# Features

* 轻量级部署：可以单独部署，也支持集群部署
* 多种类型的数据输入以及输出，source 支持 [RocketMQ](https://github.com/apache/rocketmq) ， sink 支持db, rocketmq 等
* 多种类型的数据输入以及输出;

# Architecture
- [整体架构](docs/design/1.RocketMQ-streams整体架构.md)： 介绍RocketMQ-streams总体构成；
Expand All @@ -21,31 +21,6 @@
- [Window算子解析](docs/design/5.Window算子解析.md)： 介绍有状态算子window实例化、数据处理、窗口触发过程；


# DataStream Example

```java
import org.apache.rocketmq.streams.client.transform.DataStream;

DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");

source
.fromFile("～/admin/data/text.txt",false)
.map(message->message)
.toPrint(1)
.start();
```

# Maven Repository

```xml

<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-streams-clients</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
```

# Core API

rocketmq-stream 实现了一系列高级的API，可以让用户很方便的编写流计算的程序，实现自己的业务需求；
Expand Down Expand Up @@ -123,32 +98,71 @@ DataStream实现了一系列常见的流计算算子

#### Strategy

策略机制主要用来控制计算引擎运行过程中的底层逻辑，如checkpoint，state的存储方式等，后续还会增加对窗口、双流join等的控制；所有的控制策略通过```with```算子传入，可以同时传入多个策略类型；
策略机制主要用来控制计算引擎运行过程中的底层逻辑，如window存储方式，后续还会增加对state、双流join等的控制；所有的控制策略通过```with```算子传入，可以同时传入多个策略类型；

```java
//指定checkpoint的存储策略
source
.fromRocketmq("TSG_META_INFO","")
.map(message->message+"--")
.toPrint(1)
.with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX","","",0L))
.with(WindowStrategy.highPerformance())
.start();
```

# 运行
# 构建本地运行环境

Rocketmq-Streams 作为典型的java应用，既可以集成在业务系统里运行，也可以作为一个独立的jar包来运行；
## 环境准备
- JDK 1.8及以上
- Maven 3.2及以上
- 本地启动RocketMQ，[启动文档](https://rocketmq.apache.org/docs/quick-start/)

首先对应用的源码进行编译
## 构建Rocketmq-streams

```shell
mvn -Prelease-all -DskipTests clean install -U
git clone https://github.com/apache/rocketmq-streams.git
cd rocketmq-streams
mvn clean -DskipTests install -U
```

然后直接通过java指令来运行
## pom依赖

```shell
java -jar jarName mainClass
```xml
<dependencies>
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-streams-clients</artifactId>
<!--替换成最新版本-->
<version>${version}</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<minimizeJar>false</minimizeJar>
<shadedArtifactAttached>true</shadedArtifactAttached>
<artifactSet>
<includes>
<include>org.apache.rocketmq:rocketmq-streams-clients</include>
</includes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
```

更多详细的案例可以看[这里](docs/SUMMARY.md)
## 详细的使用案例见[rocketmq-streams-examples](rocketmq-streams-examples/README.md)
87 changes: 58 additions & 29 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,47 +9,22 @@

## [中文文档](./README-chinese.md)

## [Quick Start](./quick_start.md)
## [Quick Start](quick_start.md)

## Features

* Lightweight deployment: RocketMQ Streams can be deployed separately or in cluster mode.
* Various types of data input and output: source supports [RocketMQ](https://github.com/apache/rocketmq) while sink supports databases and RocketMQ, etc.

## DataStream Example

```java
import org.apache.rocketmq.streams.client.transform.DataStream;

DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");
source
.fromFile("～/admin/data/text.txt",false)
.map(message->message)
.toPrint(1)
.start();
```

## Maven Repository

```xml

<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-streams-clients</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
```

# Core API

RocketMQ Streams implements a series of advanced APIs, allowing users to write stream computing programs conveniently and achieve their own business requirements.

## StreamBuilder

StreamBuilder is used to build the source of stream tasks. It contains two methods: ```dataStream()``` and ```tableStream()```, which return two sources, DataStreamSource and TableStreamSource, respectively.
StreamBuilder is used to build the source of stream tasks.

+ [dataStream(nameSpaceName,pipelineName)]() returns an instance of DataStreamSource, used for segmented programming to achieve stream computing tasks.
+ [tableStream(nameSpaceName,pipelineName)]() returns an instance of TableStreamSource, used for script programming to achieve stream computing tasks.

## DataStream API

Expand Down Expand Up @@ -109,15 +84,69 @@ DataStream implements a series of common stream calculation operators as follows

# Strategy

The Strategy mechanism is mainly used to control the underlying logic during the operation of the computing engine, such as the storage methods of Checkpoint and state etc. Subsequent controls for windows, dual-stream joins, and so on will be added. All control strategies are transmitted through the ```with``` operator. Multiple policy types can be transmitted at the same time.
The Strategy mechanism is mainly used to control the underlying logic during the operation of the computing engine, such as the storage methods of window. Subsequent controls for state, dual-stream joins, and so on will be added. All control strategies are transmitted through the ```with``` operator. Multiple policy types can be transmitted at the same time.

```java
//Specify the storage strategy for Checkpoint.
source
.fromRocketmq("TSG_META_INFO","")
.map(message->message+"--")
.toPrint(1)
.with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX","","",0L))
.with(WindowStrategy.highPerformance())
.start();
```

# Run local project

## environment
- JDK 1.8+
- Maven 3.2+
- Install RocketMQ in local，[intall doc](https://rocketmq.apache.org/docs/quick-start/)

## Install Rocketmq-streams

```shell
git clone https://github.com/apache/rocketmq-streams.git
cd rocketmq-streams
mvn clean -DskipTests install -U
```

## pom

```xml
<dependencies>
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-streams-clients</artifactId>
<!--Newest version-->
<version>${version}</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<minimizeJar>false</minimizeJar>
<shadedArtifactAttached>true</shadedArtifactAttached>
<artifactSet>
<includes>
<include>org.apache.rocketmq:rocketmq-streams-clients</include>
</includes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
```
142 changes: 0 additions & 142 deletions docs/README.md
View file

This file was deleted.

Loading

0 comments on commit e5fd612

Please sign in to comment.