Let snapshot saved disk same as rocksdb data path #1392

Linary · 2021-03-15T09:00:22Z

The goal is to ensure that snapshots can be generated in hard link

Change-Id: Idfa9387e46ac094a9c4cd69fee310b271127d889

hugegraph-api/src/main/java/com/baidu/hugegraph/api/profile/GraphsAPI.java

hugegraph-api/src/main/java/com/baidu/hugegraph/auth/HugeGraphAuthProxy.java

hugegraph-core/src/main/java/com/baidu/hugegraph/HugeGraph.java

javeme · 2021-03-15T09:42:31Z

...graph-core/src/main/java/com/baidu/hugegraph/backend/store/AbstractBackendStoreProvider.java

+        String snapshotPrefix = "snapshot";
+        for (BackendStore store : this.stores.values()) {
+            store.writeSnapshot(snapshotPrefix);
+        }


prefer call store.writeSnapshot() through tx, and remove writeSnapshot() from AbstractBackendStoreProvider class

Only memory backend will share store provider

hugegraph-core/src/main/java/com/baidu/hugegraph/HugeGraph.java

* The goal is to ensure that snapshots can be generated in hard link Change-Id: Idfa9387e46ac094a9c4cd69fee310b271127d889

Change-Id: I829ad9eaaf399b5fc1b8d25a90b2d1dd88c54894

Change-Id: I2b212decb8f0639354e032a09e9ca40988b001da

Change-Id: I67d88d798c6a16a66681296fbf29d73fb8bc356a

hugegraph-api/src/main/java/com/baidu/hugegraph/api/profile/GraphsAPI.java

hugegraph-core/src/main/java/com/baidu/hugegraph/StandardHugeGraph.java

...raph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftBackendStoreProvider.java

javeme · 2021-03-23T12:01:53Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftSharedContext.java

+                if (ids.size() == 1) {
+                    eventHub.call(Events.CACHE, action, type, ids.get(0));
+                } else {
+                    eventHub.call(Events.CACHE, action, type, ids.toArray());


seems can delete if-branch

Can't delete.

In CachedSchemaTransaction

if (Cache.ACTION_INVALID.equals(args[0])) { event.checkArgs(String.class, HugeType.class, Id.class);

the 3rd param must be Id class.

In CachedGraphTransaction

if (Cache.ACTION_INVALID.equals(args[0])) { event.checkArgs(String.class, HugeType.class, Object.class);

the 3rd param can be Id or Id[] class

javeme · 2021-03-23T12:02:28Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreSnapshotFile.java

@@ -48,7 +47,7 @@

    private static final Logger LOG = Log.logger(StoreSnapshotFile.class);

-    private static final String SNAPSHOT_DIR = "snapshot";
+    public static final String SNAPSHOT_DIR = "snapshot";


AbstractBackendStoreProvider has used it.

hugegraph-test/src/main/java/com/baidu/hugegraph/core/EdgeCoreTest.java

hugegraph-api/src/main/java/com/baidu/hugegraph/api/profile/GraphsAPI.java

Change-Id: I845cc78b20b94eee6bd2aff43038dd22c1a7834a

Change-Id: Iebf742426aa6b696b2397ec90368a13d04c04b56

codecov · 2021-03-24T09:29:35Z

Codecov Report

Merging #1392 (319ae66) into master (126d401) will increase coverage by 0.02%.
The diff coverage is 36.87%.

@@             Coverage Diff              @@
##             master    #1392      +/-   ##
============================================
+ Coverage     62.04%   62.07%   +0.02%     
- Complexity     5838     5853      +15     
============================================
  Files           386      386              
  Lines         32192    32290      +98     
  Branches       4497     4506       +9     
============================================
+ Hits          19975    20043      +68     
- Misses        10186    10207      +21     
- Partials       2031     2040       +9

Impacted Files	Coverage Δ	Complexity Δ
...ava/com/baidu/hugegraph/api/profile/GraphsAPI.java	`20.75% <0.00%> (-3.69%)`	`0.00 <0.00> (ø)`
...a/com/baidu/hugegraph/auth/HugeGraphAuthProxy.java	`3.91% <0.00%> (-0.05%)`	`0.00 <0.00> (ø)`
...e/src/main/java/com/baidu/hugegraph/HugeGraph.java	`65.30% <ø> (ø)`	`9.00 <0.00> (ø)`
...baidu/hugegraph/backend/store/BackendFeatures.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...h/backend/store/raft/RaftBackendStoreProvider.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ugegraph/backend/store/raft/RaftSharedContext.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ugegraph/backend/store/raft/StoreSnapshotFile.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ugegraph/backend/store/raft/StoreStateMachine.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...gegraph/backend/store/rocksdb/RocksDBFeatures.java	`78.26% <0.00%> (-3.56%)`	`18.00 <0.00> (ø)`
...gegraph/backend/store/rocksdb/RocksDBSessions.java	`100.00% <ø> (ø)`	`1.00 <0.00> (ø)`
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 126d401...319ae66. Read the comment docs.

...graph-core/src/main/java/com/baidu/hugegraph/backend/store/AbstractBackendStoreProvider.java

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreSnapshotFile.java

...raph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

javeme · 2021-03-24T11:05:24Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

+                RocksDB rocksdb = sessions.createSnapshotRocksDB(
+                                  snapshotPath.toString());
+                Path snapshotLinkPath = Paths.get(originDataPath + "_link");
+                createCheckpoint(rocksdb, snapshotLinkPath.toString());


why createCheckpoint in resumeSnapshot()

when under raft mode, need reserve the origin snapshot dir, so need create hard link from snapshot

Change-Id: Idd420fd98be6668cf0db03178530eda3b9529ea5

...raph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java

hugegraph-test/src/main/java/com/baidu/hugegraph/core/EdgeCoreTest.java

javeme · 2021-03-25T12:23:46Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

-                                               snapshotLinkDir, originDataDir);
+                Path parentPath = snapshotPath.getParent();
+                if (Files.list(parentPath).count() == 0) {
+                    FileUtils.deleteDirectory(parentPath.toFile());


seems still not use "move directory" if needed to delete snapshot

I didn't understand.

Change-Id: I29adb7df352bbbba3fbd416831b11b0968818cb0

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreSnapshotFile.java

hugegraph-core/src/main/java/com/baidu/hugegraph/security/HugeSecurityManager.java

...raph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java

Change-Id: I0561be1a71322644465e745ded9ca1636bff078a

hugegraph-core/src/main/java/com/baidu/hugegraph/security/HugeSecurityManager.java

...raph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java

javeme · 2021-03-26T10:05:54Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

@@ -614,15 +614,12 @@ protected Session session(HugeType tableType) {
            Set<String> uniqueParents = new HashSet<>();
            // Every rocksdb instance should create an snapshot
            for (Map.Entry<String, RocksDBSessions> entry : this.dbs.entrySet()) {
-                // Like: parent_path/rocksdb-data/m
-                //       parent_path/rocksdb-vertex/g
+                // Like: parent_path/rocksdb-data/*, * maybe g,m,s


maybe => can be

Change-Id: I045690d244ad83e81e7ea7624ae9388f88521142

javeme · 2021-03-26T11:00:11Z

hugegraph-core/src/main/java/com/baidu/hugegraph/security/HugeSecurityManager.java

@@ -322,7 +330,8 @@ public void checkPropertiesAccess() {
    public void checkPropertyAccess(String key) {
        if (!callFromAcceptClassLoaders() && callFromGremlin() &&
            !WHITE_SYSTEM_PROPERTYS.contains(key) && !callFromBackendHbase() &&
-            !callFromRaft() && !callFromSofaRpc()) {
+            !callFromBackendRocksDB() && !callFromRaft() &&


unnecessary?

maybe exist some dup part, but indeed need it.

...raph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

hugegraph-core/src/main/java/com/baidu/hugegraph/security/HugeSecurityManager.java

Change-Id: Ie7367816e681438dd4bc82287d1b810d781845aa

Change-Id: I70d1b3f70ac347f8e75a95d50654cf3f7f5536e8

hugegraph-api/src/main/java/com/baidu/hugegraph/api/profile/GraphsAPI.java

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreSnapshotFile.java

Change-Id: I88b39003107652f6ff8eae071e88dc888dd21959

javeme requested changes Mar 15, 2021

View reviewed changes

javeme requested changes Mar 16, 2021

View reviewed changes

hugegraph-core/src/main/java/com/baidu/hugegraph/HugeGraph.java Outdated Show resolved Hide resolved

Linary force-pushed the hardlink-snapshot branch 2 times, most recently from 9eb4eb4 to 1c8460b Compare March 22, 2021 11:20

Linary added 4 commits March 23, 2021 15:03

Let snapshot saved disk same as rocksdb data path

10b1bfe

* The goal is to ensure that snapshots can be generated in hard link Change-Id: Idfa9387e46ac094a9c4cd69fee310b271127d889

tiny improve

de8b72b

Change-Id: I829ad9eaaf399b5fc1b8d25a90b2d1dd88c54894

Add resume snapshot API

bbb6faf

Change-Id: I2b212decb8f0639354e032a09e9ca40988b001da

tiny improve

bce32ec

Change-Id: I67d88d798c6a16a66681296fbf29d73fb8bc356a

javeme requested changes Mar 23, 2021

View reviewed changes

tiny improve

5adb2c6

Change-Id: I845cc78b20b94eee6bd2aff43038dd22c1a7834a

Linary force-pushed the hardlink-snapshot branch from 1c8460b to 372657d Compare March 24, 2021 08:16

Use checkpoint when resume snapshot

74d2a33

Change-Id: Iebf742426aa6b696b2397ec90368a13d04c04b56

Linary force-pushed the hardlink-snapshot branch from 372657d to 74d2a33 Compare March 24, 2021 09:00

javeme reviewed Mar 24, 2021

View reviewed changes

some improve

084bca2

Change-Id: Idd420fd98be6668cf0db03178530eda3b9529ea5

javeme reviewed Mar 25, 2021

View reviewed changes

tiny improve

060f190

Change-Id: I29adb7df352bbbba3fbd416831b11b0968818cb0

javeme reviewed Mar 26, 2021

View reviewed changes

...raph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java Outdated Show resolved Hide resolved

Linary force-pushed the hardlink-snapshot branch from 699d03b to 607076f Compare March 26, 2021 08:42

Refactor some code in RocksDBStdSessions

fcb5860

Change-Id: I0561be1a71322644465e745ded9ca1636bff078a

Linary force-pushed the hardlink-snapshot branch from 607076f to fcb5860 Compare March 26, 2021 09:35

javeme reviewed Mar 26, 2021

View reviewed changes

tiny improve

9ffda82

Change-Id: I045690d244ad83e81e7ea7624ae9388f88521142

javeme reviewed Mar 26, 2021

View reviewed changes

tiny improve

74c0764

Change-Id: Ie7367816e681438dd4bc82287d1b810d781845aa

javeme previously approved these changes Mar 26, 2021

View reviewed changes

tiny improve

6d06009

Change-Id: I70d1b3f70ac347f8e75a95d50654cf3f7f5536e8

Linary dismissed javeme’s stale review via 6d06009 March 29, 2021 06:34

javeme previously approved these changes Mar 30, 2021

View reviewed changes

zhoney requested changes Mar 31, 2021

View reviewed changes

tiny improve

319ae66

Change-Id: I88b39003107652f6ff8eae071e88dc888dd21959

Linary dismissed javeme’s stale review via 319ae66 March 31, 2021 09:50

javeme approved these changes Apr 1, 2021

View reviewed changes

coderzc approved these changes Apr 1, 2021

View reviewed changes

zhoney approved these changes Apr 1, 2021

View reviewed changes

Linary merged commit 39b9474 into master Apr 1, 2021

Linary deleted the hardlink-snapshot branch April 1, 2021 10:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Let snapshot saved disk same as rocksdb data path #1392

Let snapshot saved disk same as rocksdb data path #1392

Linary commented Mar 15, 2021

javeme Mar 15, 2021

Linary Mar 16, 2021

javeme Mar 23, 2021

Linary Mar 24, 2021

javeme Mar 23, 2021

Linary Mar 24, 2021

codecov bot commented Mar 24, 2021 •

edited

Loading

javeme Mar 24, 2021

Linary Mar 25, 2021

javeme Mar 25, 2021 •

edited

Loading

Linary Mar 26, 2021

javeme Mar 26, 2021

javeme Mar 26, 2021

Linary Mar 26, 2021

Let snapshot saved disk same as rocksdb data path #1392

Let snapshot saved disk same as rocksdb data path #1392

Conversation

Linary commented Mar 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Mar 24, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme Mar 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Mar 24, 2021 •

edited

Loading

javeme Mar 25, 2021 •

edited

Loading