[Summary] integrate new modules PD and Store components into hugegraph (Breaking Change)

[VGalaxies]

Feature Description (功能描述)

• mailing list link: https://lists.apache.org/thread/w92mrt8b8n93yvxfqy9hvy17p355hscg
• preview release: https://github.com/hugegraph/hugegraph/releases/tag/pd-store-tmp
• deployment guide: https://github.com/apache/incubator-hugegraph/wiki/HugeGraph-Distributed-(pd%E2%80%90store)-Version-Deployment-Guide
• project tracking: https://github.com/orgs/apache/projects/316
• sub summary issue:
  • [Summary] integrate new module PD into hugegraph #2481
  • [Summary] integrate new module Store into hugegraph #2482
  • [Summary] integrate new backend Hstore into hugegraph #2483
  • [Summary] Dropping Support for Java 8: Compatibility Breaks Ahead #2499
  • [Summary] Feedback on the trial of HugeGraph Distributed (pd-store) Version (BETA) #2512

Relative Issues

#435 #828 #1218 #1398 #1517 #1581 #1609 #1760 #1925 #1968 #1979

Background

Currently, the architecture of the community version of HugeGraph is still in version 1.0. In the internal version 2.0 of HugeGraph, we have the following design goals:

• Support trillion-scale data storage and tens of thousands of graph storages.
• Support multi-active, high availability, dynamic scalability, and automated operation and maintenance.
• Maximize read and write performance.

Based on these goals, we have designed a distributed architecture that supports graph data partitioning and multiple replicas, and separates storage from computation for flexible scaling.

In version 2.0, in addition to the hugegraph-server, we also introduce two additional components: hugegraph-pd and hugegraph-store

The responsibilities of these two components are as follows:

• hugegraph-pd: pd stands for placement driver, which can be simply understood as a meta server responsible for service discovery, partition information storage, and node scheduling.
• hugegraph-store: as a new built-in storage backend, it uses RocksDB as the distributed backend storage foundation.

We will gradually merge the internal version 2.0 of HugeGraph into the community version. Therefore, the first step is to integrate the hugegraph-pd and hugegraph-store modules into this repository.

Tasks

Introduce PD and Store on the pd-store branch.

Currently, all tasks should be carried out on the pd-store branch:

• Adjust the project structure of this repository to include three sub-modules: hugegraph-server, hugegraph-pd, hugegraph-store at the root level.
  • refact: adjust hugegraph project structure to prepare for merge #2266
• Merge the internal version of pd into hugegraph.
  • refact: merge pd into hugegraph #2270
  • refact: merge pd into hugegraph (reopened) #2273
• Merge the internal version of store into hugegraph.
  • refact: merge store into hugegraph #2272
• Adjust the pom structure in order to directly build from root.
  • refact: adjust pom structure for directly compiling from root #2275
  • fix: assemble executable jar for pd and store #2289
• Configuration adjustments (including merging dependencies, unifying version numbers, Maven compilation adjustments, etc.).
  • fix: github ci #2288
  • feat: support visit of stage repository #2305
  • chore: enhance a string of problems to pass CI #2321
• Perform code adaptation tasks (hugegraph-server needs to introduce modules to connect with hugegraph-pd and hugegraph-store, and make adjustments to other modules, etc.).
  • feat: preliminary support for hstore backend #2301
  • feat: preliminary support for distributed task scheduler #2319
• Synchronization from the master branch.
  • chore: cherry-pick commits from master to pd-store for synchronization #2345
  • chore(pd-store): cherry-pick commits from master for synchronization & some modifications for passing partial hstore tests #2414

Project structure adjustments on the master branch.

After #2301, it should be possible to perform simple CRUD operations on HugeGraph on the pd-store branch. In the next phase, the pd-store branch will need to be merged into the master branch with an appropriate granularity.

• (master) project structure adjustments and CI configurations
  • refact: adjust project structure for merge PD & Store[Breaking Change] #2338
  • chore: reset hugegraph version to 1.2.0 #2382

We will reorganize commit messages and merge changes ahead of the pd-store branch into the master branch:

• pd-grpc, pd-common, pd-client (~8k loc)
• pd-core (~9k loc)
• pd-service (~11k loc)
• pd-dist, pd-test (~1k loc)

---

• store-grpc, store-common, store-client (~13k loc)
• store-rocksdb (~4k loc)
• store-core (~14k loc)
• store-node (~11k loc)
• store-dist, store-test, store-cli (~3k loc)

---

• server-hstore (~3k loc)
• server-core (~12k loc)

flowchart TD
  A(hg-pd-grpc)
  B(hg-pd-common)
  C(hg-pd-client)
  D(hugegraph-core)
  E(hugegraph-hstore)
  F(hg-store-grpc)
  G(hg-store-common)
  H(hg-store-client)
  J(hg-pd-core)
  K(hg-pd-service)
  L(hg-pd-dist)
  M(hg-store-cli)
  N(hg-store-rocksdb)
  O(hg-store-core)
  P(hg-store-node)
  Q(hg-store-dist)
  B-->A
  C-->B
  D-.->C
  E-->D
  H-->F
  H-->G
  H-->C
  D-->G
  E-->H
  O-->D
  J-->B
  K-->J
  L-->K
  M-->H
  N-->G
  O-.->H
  O-->N
  P-->O
  Q-->P

Loading

Project Structure

# new project structure
                   - core/api/test/...
          - server - hbase/rocksdb/mysql/...
          -        - ...
hugegraph - pd - pd submodules
          -
          - store - store submodules

[Pengzna]

please add store related sub-task to this issue, thx~

[VGalaxies]

close by #2575, PD and Store components have been integrated into hugegraph and will continue to evolve in future versions...