add hugegraph-ldbc

apache · Oct 6, 2023 · dd909d0 · dd909d0
1 parent 76675b4
commit dd909d0
Show file tree

Hide file tree

Showing 6 changed files with 432 additions and 0 deletions.
diff --git a/hugegraph-ldbc/README.md b/hugegraph-ldbc/README.md
@@ -0,0 +1,50 @@
+# hugegraph-ldbc
+
+此存储库包含 LDBC 社交网络基准的交互式工作负载在HugeGraph数据库的参考实现。有关LDBC基准测试的详细信息，请参阅[SIGMOD 2015 论文](https://ldbcouncil.org/docs/papers/ldbc-snb-interactive-sigmod-2015.pdf)、[GitHub Pages 上的规范](https://ldbcouncil.org/ldbc_snb_docs/)和[arXiv 上的规范](https://arxiv.org/pdf/2001.02299.pdf)。
+
+社交网络基准套件（SNB）定义了针对数据库管理系统的图形工作负载，并由LDBC SNB工作组维护，这些工作负载旨在通过不同的查询和操作模式模拟实际应用场景，对数据库管理系统进行性能测试和评估。有关LDBC SNB的详细信息，请参阅[LDBC 社交网络基准](https://docs.google.com/presentation/d/1NilxSrKQnFq4WzWMY2-OodZQ2TEksKzKBmgB20C_0Nw/)( [PDF](https://ldbcouncil.org/docs/presentations/ldbc-snb-2022-11.pdf) )。
+
+## 快速开始
+
+要构建整个项目，请运行：
+
+```
+scripts/build.sh
+```
+
+## 测试
+
+下面说明解释了如何在三种模式之一（创建验证参数、验证、基准测试）下运行基准测试驱动程序。有关驱动程序模式的更多详细信息，请查看[主自述文件的“驱动程序模式”部分](https://github.com/ldbc/ldbc_snb_interactive_v1_impls/blob/main/README.md#driver-modes)。
+
+编辑`driver/{create-validation-parameters,validate,benchmark}.properties`文件，设置正确链接HugeGraph的参数，然后运行下面：
+
+1. 创建验证参数：
+
+```Bash
+driver/create-validation-parameters.sh
+```
+
+2. 进行验证：
+
+```Bash
+driver/validate.sh
+```
+
+3. 运行基准测试：
+
+```Bash
+driver/benchmark.sh
+```
+
+注意：由于运行基准测试add操作可能会改变原本数据，所以如果重复执行基准操作时，必须重新导入数据：
+
+```Bash
+./scripts/load-in-one-step.sh 
+```
+
+详细测试测试流程请参照[HugeGraph Ldbc测试流程](https://github.com/hugegraph/ldbc/blob/lzy-patch/hugegraph/README.md)
+
+## 如何贡献
+
+直接提交pr即可
+
diff --git a/hugegraph-ldbc/ci.yml b/hugegraph-ldbc/ci.yml
@@ -0,0 +1,25 @@
+Global:
+  version: 2.0
+  #group_email: xxx@baidu.com   # <------ 配置团队邮箱地址，用于接收latest软件版本升级通知邮件
+
+Default:
+  profile: [dev]
+
+Profiles:
+  - profile:
+    name: dev
+    mode: AGENT
+    environment:
+      image: DECK_CENTOS7U5_K3         # http://buildcloud.baidu.com/submitter/5-env_declare
+      resourceType: MEDIUM
+      #tools:                          # <------ 配置软件版本信息
+      #  - go: latest                  # http://buildcloud.baidu.com/ci_yml/2-config_format_spec#20.-specify-dev-tool-for-AGENT-template
+    build:
+      command: mvn clean install -Dmaven.test.skip=true && mv target output
+      cache:
+        enable: true
+        trimeThresholdSize: 1
+        #paths:
+        #  - packages
+    artifacts:
+      release: false
diff --git a/hugegraph-ldbc/hugegraph/README.md b/hugegraph-ldbc/hugegraph/README.md
@@ -0,0 +1,145 @@
+# **A. 测试环境**
+
+推荐的环境是基准脚本（Bash）和LDBC驱动程序（Java 11）在主机上运行，而HugetGraph数据库在Docker容器中运行。因此，要求如下：
+
+- Bash
+- Java 11
+- Docker 19+
+- Maven
+- Hadoop 3.2.1
+- python 2（生成数据集时[基于 Hadoop 的 Datagen](https://github.com/ldbc/ldbc_snb_datagen_hadoop)会用到）
+
+# **B. 配置**
+
+默认环境变量（例如HugeGraph版本、容器名称等）存储在`scripts/vars.sh`. 根据需要调整这些。
+
+# **C. 生成数据**（ **利用datagen生成数据**）
+
+数据的生成是进行benchmark测试的第一步，LDBC为其制定的测试需求设计了一个数据生成器：[ldbc_snb_datagen ](https://github.com/ldbc/ldbc_snb_datagen_hadoop)，本实验使用该数据生成器进行数据生成。
+
+- 下载ldbc_snb_datagen ：
+
+  ```
+  git clone https://github.com/ldbc/ldbc_snb_datagen_hadoop.git
+  cd ldbc_snb_datagen_hadoop
+  ```
+
+  - 
+
+- 初始化params.ini文件：
+
+  ```
+  cp params-csv-basic.ini params.ini
+  ```
+
+  - 
+
+- params.ini文件配置（其中ldbc.snb.datagen.generator.scaleFactor:snb.interactive.0.1代表生成数据量大小为sf0.1，可以根据自己需要更改为0.1-1000）
+
+  ```
+  ldbc.snb.datagen.generator.scaleFactor:snb.interactive.0.1
+  ldbc.snb.datagen.serializer.dateFormatter:ldbc.snb.datagen.util.formatter.LongDateFormatter
+  ldbc.snb.datagen.serializer.dynamicActivitySerializer:ldbc.snb.datagen.serializer.snb.csv.dynamicserializer.activity.CsvCompositeDynamicActivitySerializer
+  ldbc.snb.datagen.serializer.dynamicPersonSerializer:ldbc.snb.datagen.serializer.snb.csv.dynamicserializer.person.CsvCompositeDynamicPersonSerializer
+  ldbc.snb.datagen.serializer.staticSerializer:ldbc.snb.datagen.serializer.snb.csv.staticserializer.CsvCompositeStaticSerializer
+  ```
+
+- 设置Hadoop
+
+  ```
+  wget https://archive.apache.org/dist/hadoop/core/hadoop-3.2.1/hadoop-3.2.1.tar.gz
+  tar xf hadoop-3.2.1.tar.gz
+  export HADOOP_CLIENT_OPTS="-Xmx2G"
+  # set this to the Hadoop 3.2.1 directory
+  export HADOOP_HOME=pwd/hadoop-3.2.1
+  ```
+
+  - 
+
+- 执行脚本生成数据
+
+  ```
+  ./run.sh
+  ```
+
+  - 
+
+- 将生成的数据集拷贝出来
+
+  ```
+  cd ..
+  mkdir ldbc-master/update-streams
+  cp -r ldbc_snb_datagen_hadoop-main/social_network/. ldbc-master/update-streams/
+  cp -r ldbc_snb_datagen_hadoop-main/social_network/ ldbc-master/hugegraph/
+  cp -r ldbc_snb_datagen_hadoop-main/substitution_parameters/ ldbc-master/
+  ```
+
+  - 
+
+- 对生成csv文件进行数据处理
+
+  ```
+  cd ldbc-master/hugegraph
+  ./scripts/reply_csv_column.sh
+  ```
+
+  - 
+
+# **D. 导入数据**
+
+## 1. **设置**`${HugeGraph_CSV_DIR}`**环境变量**。
+
+首先进入到ldbc-master/hugegraph/目录，再执行：
+
+```Bash
+. scripts/use-datagen-data-set.sh
+```
+
+## 2. **加载数据：**
+
+**注意：需要先将****[hugegraph-loader](https://hugegraph.apache.org/cn/docs/quickstart/hugegraph-loader/)****编译好并且拷贝到ldbc-master/hugegraph目录下**
+
+分步执行：
+
+```Bash
+./scripts/stop.sh
+./scripts/import.sh
+./scripts/start.sh
+```
+
+或者直接执行一键部署脚本：
+
+```Bash
+./scripts/load-in-one-step.sh 
+```
+
+
+# E. **运行**基准测试
+
+下面说明解释了如何在三种模式之一（创建验证参数、验证、基准测试）下运行基准测试驱动程序。有关驱动程序模式的更多详细信息，请查看[主自述文件的“驱动程序模式”部分](https://github.com/ldbc/ldbc_snb_interactive_v1_impls/blob/main/README.md#driver-modes)。
+
+编辑`driver/{create-validation-parameters,validate,benchmark}.properties`文件，设置正确链接HugeGraph的参数，然后运行下面：
+
+1. 创建验证参数：
+
+```Bash
+driver/create-validation-parameters.sh
+```
+
+1. 进行验证：
+
+```Bash
+driver/validate.sh
+```
+
+1. 运行基准测试：
+
+```Bash
+driver/benchmark.sh
+```
+
+注意：由于运行基准测试add操作可能会改变原本数据，所以如果重复执行基准操作时，必须重新导入数据：
+
+```Bash
+./scripts/load-in-one-step.sh 
+```
diff --git a/hugegraph-ldbc/hugegraph/dependency-reduced-pom.xml b/hugegraph-ldbc/hugegraph/dependency-reduced-pom.xml
@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+  <parent>
+    <artifactId>impls</artifactId>
+    <groupId>org.ldbcouncil.snb</groupId>
+    <version>1.2.0-SNAPSHOT</version>
+  </parent>
+  <modelVersion>4.0.0</modelVersion>
+  <artifactId>hugegraph</artifactId>
+  <name>Hugegraph client</name>
+  <description>Hugegraph implementation for the LDBC SNB driver</description>
+  <build>
+    <plugins>
+      <plugin>
+        <artifactId>maven-shade-plugin</artifactId>
+        <version>2.4.1</version>
+        <executions>
+          <execution>
+            <phase>package</phase>
+            <goals>
+              <goal>shade</goal>
+            </goals>
+          </execution>
+        </executions>
+        <configuration />
+      </plugin>
+    </plugins>
+  </build>
+</project>
+
diff --git a/hugegraph-ldbc/hugegraph/pom.xml b/hugegraph-ldbc/hugegraph/pom.xml
@@ -0,0 +1,68 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+  -->
+
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xmlns="http://maven.apache.org/POM/4.0.0"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>org.ldbcouncil.snb</groupId>
+        <artifactId>impls</artifactId>
+        <version>1.2.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>hugegraph</artifactId>
+    <name>Hugegraph client</name>
+    <description>Hugegraph implementation for the LDBC SNB driver</description>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.ldbcouncil.snb</groupId>
+            <artifactId>common</artifactId>
+            <version>1.2.0-SNAPSHOT</version>
+        </dependency>
+
+
+        <dependency>
+            <groupId>org.apache.hugegraph</groupId>
+            <artifactId>hugegraph-client</artifactId>
+            <version>1.0.0</version>
+        </dependency>
+
+    </dependencies>
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-shade-plugin</artifactId>
+                <version>2.4.1</version>
+                <configuration>
+                    <!-- put your configurations here -->
+                </configuration>
+                <executions>
+                    <execution>
+                        <phase>package</phase>
+                        <goals>
+                            <goal>shade</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+</project>