diff --git a/hugegraph-ldbc/README.md b/hugegraph-ldbc/README.md new file mode 100644 index 000000000..bb237dbcb --- /dev/null +++ b/hugegraph-ldbc/README.md @@ -0,0 +1,50 @@ +# hugegraph-ldbc + +此存储库包含 LDBC 社交网络基准的交互式工作负载在HugeGraph数据库的参考实现。有关LDBC基准测试的详细信息,请参阅[SIGMOD 2015 论文](https://ldbcouncil.org/docs/papers/ldbc-snb-interactive-sigmod-2015.pdf)、[GitHub Pages 上的规范](https://ldbcouncil.org/ldbc_snb_docs/)和[arXiv 上的规范](https://arxiv.org/pdf/2001.02299.pdf)。 + +社交网络基准套件(SNB)定义了针对数据库管理系统的图形工作负载,并由LDBC SNB工作组维护,这些工作负载旨在通过不同的查询和操作模式模拟实际应用场景,对数据库管理系统进行性能测试和评估。有关LDBC SNB的详细信息,请参阅[LDBC 社交网络基准](https://docs.google.com/presentation/d/1NilxSrKQnFq4WzWMY2-OodZQ2TEksKzKBmgB20C_0Nw/)( [PDF](https://ldbcouncil.org/docs/presentations/ldbc-snb-2022-11.pdf) )。 + +## 快速开始 + +要构建整个项目,请运行: + +``` +scripts/build.sh +``` + +## 测试 + +下面说明解释了如何在三种模式之一(创建验证参数、验证、基准测试)下运行基准测试驱动程序。有关驱动程序模式的更多详细信息,请查看[主自述文件的“驱动程序模式”部分](https://github.com/ldbc/ldbc_snb_interactive_v1_impls/blob/main/README.md#driver-modes)。 + +编辑`driver/{create-validation-parameters,validate,benchmark}.properties`文件,设置正确链接HugeGraph的参数,然后运行下面: + +1. 创建验证参数: + +```Bash +driver/create-validation-parameters.sh +``` + +2. 进行验证: + +```Bash +driver/validate.sh +``` + +3. 运行基准测试: + +```Bash +driver/benchmark.sh +``` + +注意:由于运行基准测试add操作可能会改变原本数据,所以如果重复执行基准操作时,必须重新导入数据: + +```Bash +./scripts/load-in-one-step.sh +``` + +详细测试测试流程请参照[HugeGraph Ldbc测试流程](https://github.com/hugegraph/ldbc/blob/lzy-patch/hugegraph/README.md) + +## 如何贡献 + +直接提交pr即可 + diff --git a/hugegraph-ldbc/ci.yml b/hugegraph-ldbc/ci.yml new file mode 100644 index 000000000..8cca39e94 --- /dev/null +++ b/hugegraph-ldbc/ci.yml @@ -0,0 +1,25 @@ +Global: + version: 2.0 + #group_email: xxx@baidu.com # <------ 配置团队邮箱地址,用于接收latest软件版本升级通知邮件 + +Default: + profile: [dev] + +Profiles: + - profile: + name: dev + mode: AGENT + environment: + image: DECK_CENTOS7U5_K3 # http://buildcloud.baidu.com/submitter/5-env_declare + resourceType: MEDIUM + #tools: # <------ 配置软件版本信息 + # - go: latest # http://buildcloud.baidu.com/ci_yml/2-config_format_spec#20.-specify-dev-tool-for-AGENT-template + build: + command: mvn clean install -Dmaven.test.skip=true && mv target output + cache: + enable: true + trimeThresholdSize: 1 + #paths: + # - packages + artifacts: + release: false \ No newline at end of file diff --git a/hugegraph-ldbc/hugegraph/README.md b/hugegraph-ldbc/hugegraph/README.md new file mode 100644 index 000000000..a61c273db --- /dev/null +++ b/hugegraph-ldbc/hugegraph/README.md @@ -0,0 +1,145 @@ +# **A. 测试环境** + +推荐的环境是基准脚本(Bash)和LDBC驱动程序(Java 11)在主机上运行,而HugetGraph数据库在Docker容器中运行。因此,要求如下: + +- Bash +- Java 11 +- Docker 19+ +- Maven +- Hadoop 3.2.1 +- python 2(生成数据集时[基于 Hadoop 的 Datagen](https://github.com/ldbc/ldbc_snb_datagen_hadoop)会用到) + +# **B. 配置** + +默认环境变量(例如HugeGraph版本、容器名称等)存储在`scripts/vars.sh`. 根据需要调整这些。 + +# **C. 生成数据**( **利用datagen生成数据**) + +数据的生成是进行benchmark测试的第一步,LDBC为其制定的测试需求设计了一个数据生成器:[ldbc_snb_datagen ](https://github.com/ldbc/ldbc_snb_datagen_hadoop),本实验使用该数据生成器进行数据生成。 + +- 下载ldbc_snb_datagen : + + ``` + git clone https://github.com/ldbc/ldbc_snb_datagen_hadoop.git + cd ldbc_snb_datagen_hadoop + ``` + + - + +- 初始化params.ini文件: + + ``` + cp params-csv-basic.ini params.ini + ``` + + - + +- params.ini文件配置(其中ldbc.snb.datagen.generator.scaleFactor:snb.interactive.0.1代表生成数据量大小为sf0.1,可以根据自己需要更改为0.1-1000) + + ``` + ldbc.snb.datagen.generator.scaleFactor:snb.interactive.0.1 + ldbc.snb.datagen.serializer.dateFormatter:ldbc.snb.datagen.util.formatter.LongDateFormatter + ldbc.snb.datagen.serializer.dynamicActivitySerializer:ldbc.snb.datagen.serializer.snb.csv.dynamicserializer.activity.CsvCompositeDynamicActivitySerializer + ldbc.snb.datagen.serializer.dynamicPersonSerializer:ldbc.snb.datagen.serializer.snb.csv.dynamicserializer.person.CsvCompositeDynamicPersonSerializer + ldbc.snb.datagen.serializer.staticSerializer:ldbc.snb.datagen.serializer.snb.csv.staticserializer.CsvCompositeStaticSerializer + ``` + +- 设置Hadoop + + ``` + wget https://archive.apache.org/dist/hadoop/core/hadoop-3.2.1/hadoop-3.2.1.tar.gz + tar xf hadoop-3.2.1.tar.gz + export HADOOP_CLIENT_OPTS="-Xmx2G" + # set this to the Hadoop 3.2.1 directory + export HADOOP_HOME=pwd/hadoop-3.2.1 + ``` + + - + +- 执行脚本生成数据 + + ``` + ./run.sh + ``` + + - + +- 将生成的数据集拷贝出来 + + ``` + cd .. + mkdir ldbc-master/update-streams + cp -r ldbc_snb_datagen_hadoop-main/social_network/. ldbc-master/update-streams/ + cp -r ldbc_snb_datagen_hadoop-main/social_network/ ldbc-master/hugegraph/ + cp -r ldbc_snb_datagen_hadoop-main/substitution_parameters/ ldbc-master/ + ``` + + - + +- 对生成csv文件进行数据处理 + + ``` + cd ldbc-master/hugegraph + ./scripts/reply_csv_column.sh + ``` + + - + +# **D. 导入数据** + +## 1. **设置**`${HugeGraph_CSV_DIR}`**环境变量**。 + +首先进入到ldbc-master/hugegraph/目录,再执行: + +```Bash +. scripts/use-datagen-data-set.sh +``` + +## 2. **加载数据:** + +**注意:需要先将****[hugegraph-loader](https://hugegraph.apache.org/cn/docs/quickstart/hugegraph-loader/)****编译好并且拷贝到ldbc-master/hugegraph目录下** + +分步执行: + +```Bash +./scripts/stop.sh +./scripts/import.sh +./scripts/start.sh +``` + +或者直接执行一键部署脚本: + +```Bash +./scripts/load-in-one-step.sh +``` + + +# E. **运行**基准测试 + +下面说明解释了如何在三种模式之一(创建验证参数、验证、基准测试)下运行基准测试驱动程序。有关驱动程序模式的更多详细信息,请查看[主自述文件的“驱动程序模式”部分](https://github.com/ldbc/ldbc_snb_interactive_v1_impls/blob/main/README.md#driver-modes)。 + +编辑`driver/{create-validation-parameters,validate,benchmark}.properties`文件,设置正确链接HugeGraph的参数,然后运行下面: + +1. 创建验证参数: + +```Bash +driver/create-validation-parameters.sh +``` + +1. 进行验证: + +```Bash +driver/validate.sh +``` + +1. 运行基准测试: + +```Bash +driver/benchmark.sh +``` + +注意:由于运行基准测试add操作可能会改变原本数据,所以如果重复执行基准操作时,必须重新导入数据: + +```Bash +./scripts/load-in-one-step.sh +``` diff --git a/hugegraph-ldbc/hugegraph/dependency-reduced-pom.xml b/hugegraph-ldbc/hugegraph/dependency-reduced-pom.xml new file mode 100644 index 000000000..98bbd8bed --- /dev/null +++ b/hugegraph-ldbc/hugegraph/dependency-reduced-pom.xml @@ -0,0 +1,30 @@ + + + + impls + org.ldbcouncil.snb + 1.2.0-SNAPSHOT + + 4.0.0 + hugegraph + Hugegraph client + Hugegraph implementation for the LDBC SNB driver + + + + maven-shade-plugin + 2.4.1 + + + package + + shade + + + + + + + + + diff --git a/hugegraph-ldbc/hugegraph/pom.xml b/hugegraph-ldbc/hugegraph/pom.xml new file mode 100644 index 000000000..da45ccedd --- /dev/null +++ b/hugegraph-ldbc/hugegraph/pom.xml @@ -0,0 +1,68 @@ + + + + 4.0.0 + + + org.ldbcouncil.snb + impls + 1.2.0-SNAPSHOT + + + hugegraph + Hugegraph client + Hugegraph implementation for the LDBC SNB driver + + + + org.ldbcouncil.snb + common + 1.2.0-SNAPSHOT + + + + + org.apache.hugegraph + hugegraph-client + 1.0.0 + + + + + + + org.apache.maven.plugins + maven-shade-plugin + 2.4.1 + + + + + + package + + shade + + + + + + + diff --git a/hugegraph-ldbc/pom.xml b/hugegraph-ldbc/pom.xml new file mode 100644 index 000000000..e6a186061 --- /dev/null +++ b/hugegraph-ldbc/pom.xml @@ -0,0 +1,114 @@ + + + + + 4.0.0 + + org.ldbcouncil.snb + impls + 1.2.0-SNAPSHOT + pom + LDBC SNB Interactive implementations + + + UTF-8 + true + + 8 + 8 + + + + common + + + + + + org.apache.maven.plugins + maven-source-plugin + 2.4 + + + attach-sources + + jar + + + + + + + + + + org.apache.maven.plugins + maven-dependency-plugin + + + copy-dependencies + package + + copy-dependencies + + + ${project.build.directory}/lib + false + false + true + + + + + + + + + + + snb-mvn + https://ldbcouncil.org/snb-mvn/ + + true + always + + + + + + + all + + true + + + hugegraph + + + + hugegraph + + hugegraph + + + + + diff --git a/hugegraph-loader/assembly/static/bin/hugegraph-loader.sh b/hugegraph-loader/assembly/static/bin/hugegraph-loader.sh index b0646aba7..fbfad9002 100755 --- a/hugegraph-loader/assembly/static/bin/hugegraph-loader.sh +++ b/hugegraph-loader/assembly/static/bin/hugegraph-loader.sh @@ -25,6 +25,7 @@ abs_path() { echo "$( cd -P "$( dirname "$SOURCE" )" && pwd )" } + BIN=$(abs_path) TOP="$(cd ${BIN}/../ && pwd)" CONF="$TOP/conf"