Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update eco tools #1057

Merged
merged 5 commits into from
Oct 18, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs-2.0/1.introduction/1.what-is-nebula-graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Nebula Graph支持严格的角色访问控制和LDAP(Lightweight Directory Acc

### 生态多样化

Nebula Graph开放了越来越多的原生工具,例如[Nebula Graph Studio](https://github.com/vesoft-inc/nebula-studio)、[Nebula Console](https://github.com/vesoft-inc/nebula-console)、[Nebula Exchange](https://github.com/vesoft-inc/nebula-spark-utils/tree/v2.0.0/nebula-exchange)等,更多工具可以查看 [生态工具概览](../20.appendix/6.eco-tool-version.md)。
Nebula Graph开放了越来越多的原生工具,例如[Nebula Graph Studio](https://github.com/vesoft-inc/nebula-studio)、[Nebula Console](https://github.com/vesoft-inc/nebula-console)、[Nebula Exchange](https://github.com/vesoft-inc/nebula-exchange)等,更多工具可以查看 [生态工具概览](../20.appendix/6.eco-tool-version.md)。

此外,Nebula Graph还具备与Spark、Flink、HBase等产品整合的能力,在这个充满挑战与机遇的时代,大大增强了自身的竞争力。

Expand Down
14 changes: 7 additions & 7 deletions docs-2.0/20.appendix/6.eco-tool-version.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Nebula Explorer(简称Explorer)是一款可以通过Web访问的图探索可

Nebula Exchange(简称Exchange)是一款Apache Spark™应用,用于在分布式环境中将集群中的数据批量迁移到Nebula Graph中,能支持多种不同格式的批式数据和流式数据的迁移。详情请参见[什么是Nebula Exchange](../nebula-exchange/about-exchange/ex-ug-what-is-exchange.md)。

|Nebula Graph版本|[Exchange](https://github.com/vesoft-inc/nebula-spark-utils/tree/{{exchange.branch}}/nebula-exchange)版本(commit id)|
|Nebula Graph版本|Exchange版本(commit id)|
|:---|:---|
| {{ nebula.release }} | {{exchange.release}}(2c61ca5) |

Expand All @@ -82,39 +82,39 @@ Nebula Importer(简称Importer)是一款Nebula Graph的CSV文件导入工具

Nebula Spark Connector是一个Spark连接器,提供通过Spark标准形式读写Nebula Graph数据的能力。Nebula Spark Connector由Reader和Writer两部分组成。详情请参见[什么是Nebula Spark Connector](../nebula-spark-connector.md)。

|Nebula Graph版本|[Spark Connector](https://github.com/vesoft-inc/nebula-spark-utils/tree/{{sparkconnector.branch}}/nebula-spark-connector)版本(commit id)|
|Nebula Graph版本|Spark Connector版本(commit id)|
|:---|:---|
| {{ nebula.release }} | {{sparkconnector.release}}(2c61ca5) |

## Nebula Flink Connector

Nebula Flink Connector是一款帮助Flink用户快速访问Nebula Graph的连接器,支持从Nebula Graph图数据库中读取数据,或者将其他外部数据源读取的数据写入Nebula Graph图数据库。详情请参见[什么是Nebula Flink Connector](../nebula-flink-connector.md)。

|Nebula Graph版本|[Flink Connector](https://github.com/vesoft-inc/nebula-flink-connector/tree/{{flinkconnector.branch}})版本(commit id)|
|Nebula Graph版本|Flink Connector版本(commit id)|
|:---|:---|
| {{ nebula.release }} | {{flinkconnector.release}}(49b8f3d) |

## Nebula Algorithm

Nebula Algorithm(简称Algorithm)是一款基于[GraphX](https://spark.apache.org/graphx/)的Spark应用程序,通过提交Spark任务的形式使用完整的算法工具对Nebula Graph数据库中的数据执行图计算,也可以通过编程形式调用lib库下的算法针对DataFrame执行图计算。详情请参见[什么是Nebula Algorithm](../nebula-algorithm.md)。

|Nebula Graph版本|[Algorithm](https://github.com/vesoft-inc/nebula-spark-utils/tree/{{algorithm.branch}}/nebula-algorithm)版本(commit id)|
|Nebula Graph版本|Algorithm版本(commit id)|
|:---|:---|
| {{ nebula.release }} | {{algorithm.release}}(2c61ca5) |

## Nebula Console

Nebula Console是Nebula Graph的原生CLI客户端。如何使用请参见[连接Nebula Graph](../2.quick-start/3.connect-to-nebula-graph.md)。

|Nebula Graph版本|[Console](https://github.com/vesoft-inc/nebula-console)版本(commit id)|
|Nebula Graph版本|Console版本(commit id)|
|:---|:---|
| {{ nebula.release }} | {{console.release}}(3ce5151) |

## Nebula Docker Compose

Docker Compose可以快速部署Nebula Graph集群。如何使用请参见[Docker Compose部署Nebula Graph](../4.deployment-and-installation/2.compile-and-install-nebula-graph/3.deploy-nebula-graph-with-docker-compose.md)。

|Nebula Graph版本|[Docker Compose](https://github.com/vesoft-inc/nebula-docker-compose/tree/master)版本(commit id)|
|Nebula Graph版本|Docker Compose版本(commit id)|
|:---|:---|
| {{ nebula.release }} | {{dockercompose.release}}(d42231f) |

Expand All @@ -132,7 +132,7 @@ Docker Compose可以快速部署Nebula Graph集群。如何使用请参见[Docke

[Nebula Bench](https://github.com/vesoft-inc/nebula-bench) 用于测试Nebula Graph的基线性能数据,使用LDBC v0.3.3的标准数据集。

|Nebula Graph版本|[Nebula Bench](https://github.com/vesoft-inc/nebula-bench)版本(commit id)|
|Nebula Graph版本|Nebula Bench版本(commit id)|
|:---|:---|
| {{ nebula.release }} | {{bench.release}}(661f871) |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ Nebula Graph历史版本指低于Nebula Graph v2.0.0-GA的版本,本文介绍

11. 升级其他客户端。

所有的客户端都必须升级到支持Nebula Graph v{{nebula.release}} 的版本。包括但不限于[Python](https://github.com/vesoft-inc/nebula-python)、[Java](https://github.com/vesoft-inc/nebula-java)、[go](https://github.com/vesoft-inc/nebula-go)、[C++](https://github.com/vesoft-inc/nebula-cpp)、[Flink-connector](https://github.com/vesoft-inc/nebula-flink-connector)、[Spark-util](https://github.com/vesoft-inc/nebula-spark-utils)、[Nebula Bench](https://github.com/vesoft-inc/nebula-bench)。请找到各 repo 对应的 branch。
所有的客户端都必须升级到支持Nebula Graph v{{nebula.release}} 的版本。包括但不限于[Python](https://github.com/vesoft-inc/nebula-python)、[Java](https://github.com/vesoft-inc/nebula-java)、[go](https://github.com/vesoft-inc/nebula-go)、[C++](https://github.com/vesoft-inc/nebula-cpp)、[Flink-connector](https://github.com/vesoft-inc/nebula-flink-connector)、[Algorithm](https://github.com/vesoft-inc/nebula-algorithm)、[Exchange](https://github.com/vesoft-inc/nebula-exchange)、[Spark-connector](https://github.com/vesoft-inc/nebula-spark-connector)、[Nebula Bench](https://github.com/vesoft-inc/nebula-bench)。请找到各 repo 对应的 branch。

!!! Note

Expand Down
23 changes: 12 additions & 11 deletions docs-2.0/nebula-algorithm.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Nebula Algorithm

[Nebula Algorithm](https://github.com/vesoft-inc/nebula-spark-utils/tree/master/nebula-algorithm) (简称Algorithm)是一款基于[GraphX](https://spark.apache.org/graphx/)的Spark应用程序,通过提交Spark任务的形式使用完整的算法工具对Nebula Graph数据库中的数据执行图计算,也可以通过编程形式调用lib库下的算法针对DataFrame执行图计算。
[Nebula Algorithm](https://github.com/vesoft-inc/nebula-algorithm) (简称Algorithm)是一款基于[GraphX](https://spark.apache.org/graphx/)的Spark应用程序,通过提交Spark任务的形式使用完整的算法工具对Nebula Graph数据库中的数据执行图计算,也可以通过编程形式调用lib库下的算法针对DataFrame执行图计算。

## 前提条件

Expand Down Expand Up @@ -47,22 +47,22 @@ Nebula Algorithm实现图计算的流程如下:

3. 调用GraphX提供的图算法(例如PageRank)或者自行实现的算法(例如Louvain社区发现)。

详细的实现方法可以参见相关[Scala文件](https://github.com/vesoft-inc/nebula-spark-utils/tree/master/nebula-algorithm/src/main/scala/com/vesoft/nebula/algorithm/lib)。
详细的实现方法可以参见相关[Scala文件](https://github.com/vesoft-inc/nebula-algorithm/tree/master/nebula-algorithm/src/main/scala/com/vesoft/nebula/algorithm/lib)。

## 获取Nebula Algorithm

### 编译打包

1. 克隆仓库`nebula-spark-utils`。
1. 克隆仓库`nebula-algorithm`。

```bash
$ git clone -b {{algorithm.branch}} https://github.com/vesoft-inc/nebula-spark-utils.git
$ git clone -b {{algorithm.branch}} https://github.com/vesoft-inc/nebula-algorithm.git
```

2. 进入目录`nebula-algorithm`。

```bash
$ cd nebula-spark-utils/nebula-algorithm
$ cd nebula-algorithm
```

3. 编译打包。
Expand All @@ -87,15 +87,16 @@ Nebula Algorithm实现图计算的流程如下:

```bash
<dependency>
<groupId>com.vesoft</groupId>
<artifactId>nebula-algorithm</artifactId>
<version>{{algorithm.release}}</version>
<groupId>com.vesoft</groupId>
<artifactId>nebula-algorithm</artifactId>
<version>{{algorithm.release}}</version>
</dependency>
```

2. 传入参数调用算法(以PageRank为例)。更多算法请参见[测试用例](https://github.com/vesoft-inc/nebula-spark-utils/tree/master/nebula-algorithm/src/test/scala/com/vesoft/nebula/algorithm/lib)。
2. 传入参数调用算法(以PageRank为例)。更多算法请参见[测试用例](https://github.com/vesoft-inc/nebula-algorithm/tree/master/nebula-algorithm/src/test/scala/com/vesoft/nebula/algorithm/lib)。

!!! note

执行算法的DataFrame默认第一列是起始点,第二列是目的点,第三列是边权重(非Nebula Graph中的Rank)。

```bash
Expand All @@ -108,7 +109,7 @@ Nebula Algorithm实现图计算的流程如下:
!!! note
使用封装好的算法包有一定的局限性,例如落库到Nebula Graph时,落库的图空间中创建的Tag的属性名称必须和代码内预设的名称保持一致。如果用户有开发能力,推荐使用第一种方法。

1. 设置[配置文件](https://github.com/vesoft-inc/nebula-spark-utils/blob/{{algorithm.branch}}/nebula-algorithm/src/main/resources/application.conf)。
1. 设置[配置文件](https://github.com/vesoft-inc/nebula-algorithm/blob/{{algorithm.branch}}/nebula-algorithm/src/main/resources/application.conf)。

```bash
{
Expand Down Expand Up @@ -265,7 +266,7 @@ Nebula Algorithm实现图计算的流程如下:
示例:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.algorithm.Main /root/nebula-spark-utils/nebula-algorithm/target/nebula-algorithm-{{algorithm.release}}.jar -p /root/nebula-spark-utils/nebula-algorithm/src/main/resources/application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.algorithm.Main /root/nebula-algorithm/target/nebula-algorithm-{{algorithm.release}}.jar -p /root/nebula-algorithm/src/main/resources/application.conf
```

## 视频
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 什么是Nebula Exchange

[Nebula Exchange](https://github.com/vesoft-inc/nebula-spark-utils/tree/{{exchange.branch}}/nebula-exchange)(简称Exchange)是一款Apache Spark&trade;应用,用于在分布式环境中将集群中的数据批量迁移到Nebula Graph中,能支持多种不同格式的批式数据和流式数据的迁移。
[Nebula Exchange](https://github.com/vesoft-inc/nebula-exchange)(简称Exchange)是一款Apache Spark&trade;应用,用于在分布式环境中将集群中的数据批量迁移到Nebula Graph中,能支持多种不同格式的批式数据和流式数据的迁移。

Exchange由Reader、Processor和Writer三部分组成。Reader读取不同来源的数据返回DataFrame后,Processor遍历DataFrame的每一行,根据配置文件中`fields`的映射关系,按列名获取对应的值。在遍历指定批处理的行数后,Writer会将获取的数据一次性写入到Nebula Graph中。下图描述了Exchange完成数据转换和迁移的过程。

Expand Down
8 changes: 4 additions & 4 deletions docs-2.0/nebula-exchange/ex-ug-compile.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,16 @@

## 编译Exchange

1. 在根目录克隆仓库`nebula-spark-utils`。
1. 在根目录克隆仓库`nebula-exchange`。

```bash
git clone -b {{exchange.branch}} https://github.com/vesoft-inc/nebula-spark-utils.git
git clone -b {{exchange.branch}} https://github.com/vesoft-inc/nebula-exchange.git
```

2. 切换到目录`nebula-exchange`。

```bash
cd nebula-spark-utils/nebula-exchange
cd nebula-exchange/nebula-exchange
```

3. 打包Nebula Exchange。
Expand Down Expand Up @@ -55,7 +55,7 @@
!!! note
JAR文件版本号会因Nebula Java Client的发布版本而变化。用户可以在[Releases页面](https://github.com/vesoft-inc/nebula-java/releases)查看最新版本。

迁移数据时,用户可以参考配置文件[`target/classes/application.conf`](https://github.com/vesoft-inc/nebula-spark-utils/blob/master/nebula-exchange/src/main/resources/application.conf)。
迁移数据时,用户可以参考配置文件[`target/classes/application.conf`](https://github.com/vesoft-inc/nebula-exchange/blob/master/nebula-exchange/src/main/resources/application.conf)。

## 下载依赖包失败

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 配置说明

本文介绍使用Nebula Exchange时如何修改配置文件[`application.conf`](https://github.com/vesoft-inc/nebula-spark-utils/blob/master/nebula-exchange/src/main/resources/application.conf)。
本文介绍使用Nebula Exchange时如何修改配置文件[`application.conf`](https://github.com/vesoft-inc/nebula-exchange/blob/master/nebula-exchange/src/main/resources/application.conf)。

修改配置文件之前,建议根据数据源复制并修改文件名称,便于区分。例如数据源为CSV文件,可以复制为`csv_application.conf`。

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
示例:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/clickhouse_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/clickhouse_application.conf
```

用户可以在返回信息中搜索`batchSuccess.<tag_name/edge_name>`,确认成功的数量。例如`batchSuccess.follow: 300`。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -379,7 +379,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
示例:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/csv_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/csv_application.conf
```

用户可以在返回信息中搜索`batchSuccess.<tag_name/edge_name>`,确认成功的数量。例如`batchSuccess.follow: 300`。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
示例:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/hbase_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/hbase_application.conf
```

用户可以在返回信息中搜索`batchSuccess.<tag_name/edge_name>`,确认成功的数量。例如`batchSuccess.follow: 300`。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -341,7 +341,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
示例:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/hive_application.conf -h
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/hive_application.conf -h
```

用户可以在返回信息中搜索`batchSuccess.<tag_name/edge_name>`,确认成功的数量。例如`batchSuccess.follow: 300`。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
示例:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/json_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-echange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/json_application.conf
```

用户可以在返回信息中搜索`batchSuccess.<tag_name/edge_name>`,确认成功的数量。例如`batchSuccess.follow: 300`。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
示例:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/kafka_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/kafka_application.conf
```

用户可以在返回信息中搜索`batchSuccess.<tag_name/edge_name>`,确认成功的数量。例如`batchSuccess.follow: 300`。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
示例:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/maxcompute_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/maxcompute_application.conf
```

用户可以在返回信息中搜索`batchSuccess.<tag_name/edge_name>`,确认成功的数量。例如`batchSuccess.follow: 300`。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@ ${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchan
示例:

```bash
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-spark-utils/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-spark-utils/nebula-exchange/target/classes/mysql_application.conf
${SPARK_HOME}/bin/spark-submit --master "local" --class com.vesoft.nebula.exchange.Exchange /root/nebula-exchange/nebula-exchange/target/nebula-exchange-{{exchange.release}}.jar -c /root/nebula-exchange/nebula-exchange/target/classes/mysql_application.conf
```

用户可以在返回信息中搜索`batchSuccess.<tag_name/edge_name>`,确认成功的数量。例如`batchSuccess.follow: 300`。
Expand Down
Loading