Add system schema store #1891

zyxxoo · 2022-05-20T07:12:00Z

Add system schema store
Add meta table into system store
Add test case for read version
Let AbstractBackendStore impl default storedVersion()

codecov · 2022-05-20T07:36:59Z

Codecov Report

Merging #1891 (77e2d49) into master (1118418) will increase coverage by 0.05%.
The diff coverage is 84.09%.

@@             Coverage Diff              @@
##             master    #1891      +/-   ##
============================================
+ Coverage     71.02%   71.08%   +0.05%     
+ Complexity      982      963      -19     
============================================
  Files           451      452       +1     
  Lines         38206    38405     +199     
  Branches       5407     5418      +11     
============================================
+ Hits          27136    27299     +163     
- Misses         8420     8447      +27     
- Partials       2650     2659       +9

Impacted Files	Coverage Δ
...e/src/main/java/com/baidu/hugegraph/HugeGraph.java	`65.30% <ø> (ø)`
...du/hugegraph/backend/cache/CachedBackendStore.java	`15.15% <0.00%> (-0.48%)`	⬇️
...om/baidu/hugegraph/backend/store/BackendStore.java	`79.54% <ø> (ø)`
...m/baidu/hugegraph/backend/tx/GraphTransaction.java	`80.14% <ø> (ø)`
...n/java/com/baidu/hugegraph/config/CoreOptions.java	`99.46% <ø> (-0.02%)`	⬇️
...n/java/com/baidu/hugegraph/schema/VertexLabel.java	`86.95% <ø> (ø)`
...c/main/java/com/baidu/hugegraph/cmd/InitStore.java	`0.00% <0.00%> (ø)`
...a/com/baidu/hugegraph/auth/HugeGraphAuthProxy.java	`57.20% <10.00%> (-1.18%)`	⬇️
.../hugegraph/backend/store/AbstractBackendStore.java	`80.00% <66.66%> (-3.34%)`	⬇️
...aidu/hugegraph/backend/store/raft/RaftContext.java	`80.95% <66.66%> (-0.30%)`	⬇️
... and 49 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1118418...77e2d49. Read the comment docs.

Add meta table into system store Add test case for read version Let AbstractBackendStore impl default storedVersion() fix small bug og HugeProject add test for SystemSchemaStore Change-Id: I5c9d65c4ae022cb84592e8f2aa57bd7485cd12e4

...graph-core/src/main/java/com/baidu/hugegraph/backend/store/AbstractBackendStoreProvider.java

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/BackendStoreProvider.java

javeme · 2022-05-30T02:32:58Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/SchemaTransaction.java

@@ -70,12 +72,16 @@
 public class SchemaTransaction extends IndexableTransaction {

    private final SchemaIndexTransaction indexTx;
+    private final SystemSchemaStore systemSchemaStore;
+    private final LocalCounter counter;


TODO: move LocalCounter counter define into SystemSchemaStore class

LocalCounter provider by HugegraphParams, if we move it to SystemSchemaStore, maybe change too much code

hugegraph-dist/src/main/java/com/baidu/hugegraph/cmd/InitStore.java

javeme · 2022-05-30T02:45:45Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/AbstractBackendStore.java

@@ -25,9 +25,11 @@
 public abstract class AbstractBackendStore<Session extends BackendSession>
                implements BackendStore {

+    private final SystemSchemaStore systemSchemaStore;


TODO: move SystemSchemaStore into backend like MetaStore

this TODO annotation it first

javeme · 2022-05-30T02:47:40Z

hugegraph-core/src/main/java/com/baidu/hugegraph/StandardHugeGraph.java

+    public BackendStoreInfo backendStoreInfo() {
+        // Just for trigger Tx.getOrNewTransaction, then load 3 stores
+        this.systemTransaction();
+        return new BackendStoreInfo(this.configuration, this.storeProvider);


TODO: pass storeProvider.metaStore()

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/SchemaTransaction.java

javeme · 2022-05-30T15:06:12Z

mysql ci:

Error:  Tests run: 728, Failures: 0, Errors: 2, Skipped: 36, Time elapsed: 372.603 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
Error:  testCreateGraphsWithSameName(com.baidu.hugegraph.core.MultiGraphsTest)  Time elapsed: 0.436 s  <<< ERROR!
com.baidu.hugegraph.backend.BackendException: Failed to insert driver version with 'INSERT INTO m VALUES ('VERSION', '1.11')'
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphsWithSameName(MultiGraphsTest.java:263)

Error:  testWriteAndReadVersion(com.baidu.hugegraph.core.MultiGraphsTest)  Time elapsed: 0.206 s  <<< ERROR!
com.baidu.hugegraph.backend.BackendException: Failed to insert driver version with 'INSERT INTO m VALUES ('VERSION', '1.11')'
	at com.baidu.hugegraph.core.MultiGraphsTest.testWriteAndReadVersion(MultiGraphsTest.java:65)

hugegraph-mysql/src/main/java/com/baidu/hugegraph/backend/store/mysql/MysqlTables.java

javeme · 2022-05-30T15:22:45Z

hugegraph-mysql/src/main/java/com/baidu/hugegraph/backend/store/mysql/MysqlTables.java

+                                          driverVersion);
+            try {
+                session.execute(insert);
+            } catch (SQLException throwables) {


rename to SQLException e and pass into BackendException

method sign is session.execute() throw SQLException, so we can't change to BackendException

javeme · 2022-05-30T15:42:16Z

...-postgresql/src/main/java/com/baidu/hugegraph/backend/store/postgresql/PostgresqlTables.java

+                                          versionColumn);
+            try {
+                session.execute(insert);
+            } catch (SQLException throwables) {


javeme · 2022-05-31T01:35:32Z

ci error during init-store:

Error: Exception in thread "main" java.lang.IllegalArgumentException: Undefined vertex label: '~user'
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:163)
	at com.baidu.hugegraph.util.E.checkArgument(E.java:56)
	at com.baidu.hugegraph.StandardHugeGraph.vertexLabel(StandardHugeGraph.java:793)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.lambda$vertexLabel$2(HugeGraphAuthProxy.java:259)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.lambda$verifySchemaPermission$24(HugeGraphAuthProxy.java:914)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifyResPermission(HugeGraphAuthProxy.java:939)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifyResPermission(HugeGraphAuthProxy.java:925)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifySchemaPermission(HugeGraphAuthProxy.java:912)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifySchemaPermission(HugeGraphAuthProxy.java:905)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.vertexLabel(HugeGraphAuthProxy.java:258)
	at com.baidu.hugegraph.auth.EntityManager.queryEntity(EntityManager.java:154)
	at com.baidu.hugegraph.auth.EntityManager.query(EntityManager.java:136)
	at com.baidu.hugegraph.auth.StandardAuthManager.findUser(StandardAuthManager.java:177)
	at com.baidu.hugegraph.auth.StandardAuthenticator.requireInitAdminUser(StandardAuthenticator.java:77)
	at com.baidu.hugegraph.auth.StandardAuthenticator.initAdminUser(StandardAuthenticator.java:52)
	at com.baidu.hugegraph.auth.StandardAuthenticator.initAdminUserIfNeeded(StandardAuthenticator.java:191)
	at com.baidu.hugegraph.cmd.InitStore.main(InitStore.java:91)

hugegraph-core/src/main/java/com/baidu/hugegraph/auth/AuthManager.java

Linary · 2022-05-31T15:17:46Z

hugegraph-test/src/main/java/com/baidu/hugegraph/api/EdgeApiTest.java

 import org.junit.Before;
 import org.junit.Test;

 import com.baidu.hugegraph.testutil.Assert;
 import com.google.common.collect.ImmutableMap;

+import jakarta.ws.rs.core.Response;


seems can remove

it had been used, so we can't remove

hugegraph-test/src/main/java/com/baidu/hugegraph/unit/UnitTestSuite.java

...raph-cassandra/src/main/java/com/baidu/hugegraph/backend/store/cassandra/CassandraStore.java

hugegraph-core/src/main/java/com/baidu/hugegraph/HugeGraph.java

hugegraph-core/src/main/java/com/baidu/hugegraph/schema/VertexLabel.java

hugegraph-mysql/src/main/java/com/baidu/hugegraph/backend/store/mysql/MysqlStore.java

javeme · 2022-06-01T02:22:27Z

hugegraph-api/src/main/java/com/baidu/hugegraph/auth/HugeGraphAuthProxy.java

@@ -998,6 +998,12 @@ public HugeGraph graph() {
            return this.taskScheduler.graph();
        }

+        @Override
+        public void init() {
+            verifyAdminPermission();


this.verifyAdminPermission()

this method doesn't use this, the code is in inner class TaskSchedulerProxy

javeme · 2022-06-01T02:22:34Z

hugegraph-api/src/main/java/com/baidu/hugegraph/auth/HugeGraphAuthProxy.java

@@ -1161,6 +1167,12 @@ private String currentUsername() {
            return null;
        }

+        @Override
+        public void init() {
+            verifyAdminPermission();


this is inner class AuthManagerProxy, so can't use this.

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

javeme · 2022-06-01T17:11:58Z

...raph-cassandra/src/main/java/com/baidu/hugegraph/backend/store/cassandra/CassandraStore.java

            String driverVersion = this.provider().driverVersion();
-            this.meta.writeVersion(session, driverVersion);
+            this.meta.writeVersion(this.session(), driverVersion);


remove "this.checkOpened()" since called in this.session()

javeme · 2022-06-01T17:13:57Z

hugegraph-hbase/src/main/java/com/baidu/hugegraph/backend/store/hbase/HbaseStore.java

@@ -578,7 +574,7 @@ protected List<String> tableNames() {
        public void init() {
            super.init();
            super.checkOpened();
-            Session session = super.getSession();
+            Session session = super.sessions.session();


can call this.session(null), and we can also replace the following code with
Session session = this.session(null) like line 228,477:

this.checkOpened(); Session session = this.sessions.session();

javeme · 2022-06-01T17:17:35Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

@@ -1118,8 +1114,7 @@ public synchronized void init() {
            Lock writeLock = this.storeLock().writeLock();
            writeLock.lock();
            try {
-                super.checkOpened();
-                Session session = super.getSession();
+                Session session = super.sessions.session();


can call this.session(HugeType.META)

javeme · 2022-06-02T06:30:00Z

...raph-cassandra/src/main/java/com/baidu/hugegraph/backend/store/cassandra/CassandraStore.java

@@ -800,14 +795,13 @@ public void init() {
            super.init();
            this.checkOpened();
            String driverVersion = this.provider().driverVersion();
-            this.meta.writeVersion(this.session(), driverVersion);
+            this.meta.writeVersion(this.session(null), driverVersion);


remove this.checkOpened() call.
also update other unused call "checkOpened()"?

javeme · 2022-06-02T06:31:55Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

@@ -1129,7 +1129,7 @@ public String storedVersion() {
            readLock.lock();
            try {
                super.checkOpened();
-                Session session = super.sessions.session();
+                Session session = super.session(null);


remove this.checkOpened() call

javeme · 2022-06-02T06:32:30Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

@@ -1114,7 +1114,7 @@ public synchronized void init() {
            Lock writeLock = this.storeLock().writeLock();
            writeLock.lock();
            try {
-                Session session = super.sessions.session();
+                Session session = super.session(HugeType.META);


this.session(HugeType.META);

javeme · 2022-06-02T06:32:38Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

@@ -1129,7 +1129,7 @@ public String storedVersion() {
            readLock.lock();
            try {
                super.checkOpened();
-                Session session = super.sessions.session();
+                Session session = super.session(null);


keep this.session(HugeType.META)

javeme · 2022-06-02T06:33:55Z

hugegraph-hbase/src/main/java/com/baidu/hugegraph/backend/store/hbase/HbaseStore.java

@@ -573,17 +573,15 @@ protected List<String> tableNames() {
        @Override
        public void init() {
            super.init();
-            super.checkOpened();
-            Session session = super.sessions.session();
+            Session session = super.session(null);


keep this.session(null) style?

also update other "super.session("?

javeme · 2022-06-02T06:41:12Z

hugegraph-api/src/main/java/com/baidu/hugegraph/auth/HugeGraphAuthProxy.java

@@ -998,6 +998,12 @@ public HugeGraph graph() {
            return this.taskScheduler.graph();
        }

+        @Override
+        public void init() {
+            verifyAdminPermission();


javeme · 2022-06-02T06:43:33Z

...esql/src/main/java/com/baidu/hugegraph/backend/store/postgresql/PostgresqlStoreProvider.java

+        @Override
+        public void init() {
+            super.init();
+            this.checkOpened();


zyxxoo requested review from javeme and zhoney and removed request for javeme May 20, 2022 07:12

zyxxoo force-pushed the zy_system_store branch 2 times, most recently from ace9831 to de95390 Compare May 20, 2022 07:24

Add system schema store

c887b6b

Add meta table into system store Add test case for read version Let AbstractBackendStore impl default storedVersion() fix small bug og HugeProject add test for SystemSchemaStore Change-Id: I5c9d65c4ae022cb84592e8f2aa57bd7485cd12e4

javeme reviewed May 30, 2022

View reviewed changes

fix: remove schema failed

3e332b4

zyxxoo force-pushed the zy_system_store branch from de95390 to 3e332b4 Compare May 30, 2022 12:53

javeme reviewed May 30, 2022

View reviewed changes

zyxxoo force-pushed the zy_system_store branch from bc5b071 to c8781e8 Compare May 30, 2022 15:38

javeme reviewed May 30, 2022

View reviewed changes

fix: duplicate key error

79603ae

zyxxoo force-pushed the zy_system_store branch from c8781e8 to 79603ae Compare May 30, 2022 15:44

zyxxoo added 2 commits May 31, 2022 11:32

fix: initStore schema missed

25c5bcb

chore: improve code

7e2d3a4

Linary requested changes May 31, 2022

View reviewed changes

javeme reviewed Jun 1, 2022

View reviewed changes

improve code

1c9f0b4

zyxxoo force-pushed the zy_system_store branch from 2445d75 to 1c9f0b4 Compare June 1, 2022 16:09

javeme reviewed Jun 1, 2022

View reviewed changes

improve code

77e2d49

javeme reviewed Jun 2, 2022

View reviewed changes

Linary approved these changes Jun 2, 2022

View reviewed changes

javeme approved these changes Jun 2, 2022

View reviewed changes

javeme merged commit 2af01bc into master Jun 2, 2022

javeme deleted the zy_system_store branch June 2, 2022 08:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add system schema store #1891

Add system schema store #1891

zyxxoo commented May 20, 2022 •

edited by javeme

Loading

codecov bot commented May 20, 2022 •

edited

Loading

javeme May 30, 2022

zyxxoo May 31, 2022

javeme May 30, 2022

zyxxoo May 31, 2022

javeme May 30, 2022

javeme commented May 30, 2022

javeme May 30, 2022

zyxxoo May 31, 2022

javeme May 30, 2022

javeme commented May 31, 2022

Linary May 31, 2022

zyxxoo Jun 1, 2022

javeme Jun 1, 2022

zyxxoo Jun 1, 2022

javeme Jun 2, 2022

javeme Jun 1, 2022

zyxxoo Jun 1, 2022

javeme Jun 1, 2022

javeme Jun 1, 2022

javeme Jun 1, 2022

javeme Jun 2, 2022

javeme Jun 2, 2022

javeme Jun 2, 2022

javeme Jun 2, 2022

javeme Jun 2, 2022

javeme Jun 2, 2022

javeme Jun 2, 2022

Add system schema store #1891

Add system schema store #1891

Conversation

zyxxoo commented May 20, 2022 • edited by javeme Loading

codecov bot commented May 20, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme commented May 30, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme commented May 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zyxxoo commented May 20, 2022 •

edited by javeme

Loading

codecov bot commented May 20, 2022 •

edited

Loading