Index Handling Regression in 3.1.x / Commercial Support Discontinued?

[Jotschi]

OrientDB Version: 3.1.3

Java Version: 1.8

OS: Linux

Expected behavior

The index should at all times be in sync with the graph and a index lookup should never return null when hasNext indicated that element are available.

Actual behavior

Index lookups via tx.getEdges return null values via Iterator#next() although it.hasNext() was previously true.

Steps to reproduce

A reproducer for the issue can be found here:
https://github.com/gentics/orientdb-reproducers/blob/master/src/test/java/com/gentics/odb/IndexRegressionTest.java

I also posted details to a connected / similar issue in the commercial support system:
https://www.prjhub.com/#/issues/10986

I have however not received feedback in a long time. Is the commercial professional support still operational? We noticed that the website was taken down and all links to documentation etc. are now broken: https://orientdb.com/
Emails are also rejected by the mailserver. We did not receive any update or heads-up about these changes. Is OrientDB going out of business?

[Jotschi]

The problem can also reproduced without composite key and with Tinkerpop 3 API we get a NoSuchElementException (although hasNext indicates true).

[luigidellaquila]

Hi @Jotschi

Thank you very much for reporting.
I'm switching to private channels about the professional support (anyway, I'd say just technical problems...)

Thanks

Luigi

[luigidellaquila]

Hi @Jotschi

Thank you very much for reporting.
I'm switching to private channels about the professional support (anyway, I'd say just technical problems...)

Thanks

Luigi

[luigidellaquila]

Hi @Jotschi

As already reported on the private channels, we have a fix for this issue.
I already pushed it to 3.1.x branch and it will be released with v 3.1.4

Thanks

Luigi

[Jotschi]

@luigidellaquila Thanks for the fix. I can confirm that it fixes the reported issue. We have however still failing tests which seem to be index related. It seems the reproducer was not able to catch all issues. I'm currently working on a reproducer which also reproduces another NPE which we are having with 3.1.3+Patch.

[Jotschi]

@luigidellaquila Here is the reproducer for the second issue:
https://github.com/gentics/orientdb-reproducers/blob/master/src/test/java/com/gentics/odb/IndexRegression2Test.java

Graph is created in one tx1
Vertices are deleted in tx2 and this would need to affect the edge index as well since deleting the vertices also removes the edges - Index lookup in same tx2 fails.

[luigidellaquila]

Hi @Jotschi

I just pushed a new fix for the second test case, now hopefully all the cases should be covered.
Please feel free to reopen in case you see further corner cases to fix

Thanks!

Luigi

[Jotschi]

@luigidellaquila I applied the fix. It lets the reproducer pass but it seems the fix created a new regression.

Reproducer:
https://github.com/gentics/orientdb-reproducers/blob/master/src/test/java/com/gentics/odb/IndexRegression3Test.java

The IndexRegression3Test passed with unpatched OrientDB 3.1.3 but fails with the included patches (patches are in the project).

The test is basically the same as IndexRegression2Test. The only difference is that edges are not being removed and the test asserts that the index lookup returns the existing edges. With the provided patches this is no longer the case.

[luigidellaquila]

@Jotschi

We did a more structural change that solves this case and should also cover all the similar situations.
The fix is already on branch 3.1.x, now I'm porting it to develop

Thanks

Luigi

[Jotschi]

@luigidellaquila Thanks. The change fixed the issue. Unfortunately there are still 62 tests in our testsuite failing after upgrade to OrientDB 3.1.3 + patches. I'll investigate and create a reproducer if possible. You may close the issue for now.

[luigidellaquila]

Hi @Jotschi

OK, I'm closing this issue then.

Thanks

Luigi

[Jotschi]

@luigidellaquila I was able to create a reproducer for the next index issue:

https://github.com/gentics/orientdb-reproducers/blob/master/src/test/java/com/gentics/odb/IndexRegression5Test.java

The test creates a small tree shaped graph.
The tree is traversed via lookup of indexed vertices (NodeImpl_parent). Each vertex contains the the parent uuid. We store this within a EMBEDDEDSET, STRING property. We substituted the edges for indexed properties to reduce contention and avoid cluster issues.

Each vertex in the tree also has an outgoing edge to a ContentImpl vertex.

Traversing works fine util one of the outgoing edges gets deleted. This operation somehow breaks the NodeImpl_parent index.
The assertions in the delete(OrientGraph, String) show the problem. If you remove the e.remove(); call the test will pass.
I also validated this test agains OrientDB 3.0.34.

If I change the property type from EMBEDDEDSET, STRING to just STRING I don't get the issue. So I think it must be related to the EMBEDDEDSET handling in OrientDB.

[luigidellaquila]

Hi @Jotschi

I just pushed a fix for this problem, please let me know if there are other things to check

Thanks

Luigi

[Jotschi]

@luigidellaquila Thanks. The fix was able to reduce the test failure count to 2. I'm now checking the remaining test failures.

[Jotschi]

I found the other issue. Should be an easy fix. The problem is not related to indices thus closing this issue.