java.lang.ArrayIndexOutOfBoundsException: 64. Bulk delete vertex error

[gianluca-valentini]

OrientDB Version: 3.1.17

Java Version: 8

OS: Linux

Using OrientDB gui I need to delete a lot of vertex using delete query but I have a strange error

This is my orientdb server log trace (Analytics is my database name)

Exception 33BEB057in storageplocal:/orientdb/databases/Analytics: 3.1.17 - Veloce (build 1b0146ad0a560a69f7eb7a157843718f8d9a5440, branch UNKNOWN) com.orientechnologies.orient.core.exception.OStorageException: Internal error happened in storage Analytics please restart the server or re-open the storage to undergo the restore process and fix the error. DB name="Analytics" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkErrorState(OAbstractPaginatedStorage.java:4769) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkOpenness(OAbstractPaginatedStorage.java:4760) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.getClusterIdByName(OAbstractPaginatedStorage.java:2068) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.getClusterIdByName(ODatabaseDocumentAbstract.java:805) at com.orientechnologies.orient.core.metadata.OMetadataDefault.init(OMetadataDefault.java:129) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.loadMetadata(ODatabaseDocumentEmbedded.java:332) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.init(ODatabaseDocumentEmbedded.java:210) at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthenticate(OrientDBEmbedded.java:432) at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthenticate(OrientDBEmbedded.java:79) at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:983) at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:954) at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeDatabaseOpen37(OConnectionBinaryExecutor.java:1371) at com.orientechnologies.orient.client.remote.message.OOpen37Request.execute(OOpen37Request.java:75) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:355) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:239) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:67) Caused by: com.orientechnologies.orient.core.exception.OStorageException: Exception during execution of component operation inside of storage Analytics DB name="Analytics" at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideComponentOperation(OAtomicOperationsManager.java:242) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideComponentOperation(OAtomicOperationsManager.java:229) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.calculateInsideComponentOperation(ODurableComponent.java:96) at com.orientechnologies.orient.core.storage.index.sbtree.singlevalue.v3.CellBTreeSingleValueV3.remove(CellBTreeSingleValueV3.java:492) at com.orientechnologies.orient.core.index.engine.v1.OCellBTreeSingleValueIndexEngine.remove(OCellBTreeSingleValueIndexEngine.java:130) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.removeKeyFromIndexInternal(OAbstractPaginatedStorage.java:3029) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.removeKeyFromIndex(OAbstractPaginatedStorage.java:2991) at com.orientechnologies.orient.core.index.OIndexAbstract.doRemove(OIndexAbstract.java:628) at com.orientechnologies.orient.core.index.OIndexAbstract.doRemove(OIndexAbstract.java:602) at com.orientechnologies.orient.core.index.OIndexAbstractDelegate.doRemove(OIndexAbstractDelegate.java:442) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.applyTxChanges(OAbstractPaginatedStorage.java:2487) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitIndexes(OAbstractPaginatedStorage.java:2467) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:2382) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:2192) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.internalCommit(ODatabaseDocumentAbstract.java:2479) at com.orientechnologies.orient.core.storage.impl.local.OMicroTransaction.doCommit(OMicroTransaction.java:357) at com.orientechnologies.orient.core.storage.impl.local.OMicroTransaction.commit(OMicroTransaction.java:173) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.endMicroTransaction(ODatabaseDocumentEmbedded.java:928) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.executeDeleteRecord(ODatabaseDocumentEmbedded.java:918) at com.orientechnologies.orient.core.tx.OTransactionNoTx.deleteRecord(OTransactionNoTx.java:246) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.delete(ODatabaseDocumentEmbedded.java:1103) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.delete(ODatabaseDocumentEmbedded.java:120) at com.orientechnologies.orient.core.record.ORecordAbstract.delete(ORecordAbstract.java:288) at com.orientechnologies.orient.core.record.impl.OVertexDocument.delete(OVertexDocument.java:358) at com.orientechnologies.orient.core.record.impl.OVertexDocument.delete(OVertexDocument.java:31) at com.orientechnologies.orient.core.sql.executor.DeleteStep$1.next(DeleteStep.java:35) at com.orientechnologies.orient.core.sql.executor.CountStep.syncPull(CountStep.java:52) at com.orientechnologies.orient.core.sql.executor.OSelectExecutionPlan.fetchNext(OSelectExecutionPlan.java:34) at com.orientechnologies.orient.core.sql.executor.OUpdateExecutionPlan.executeInternal(OUpdateExecutionPlan.java:41) at com.orientechnologies.orient.core.sql.parser.ODeleteVertexStatement.execute(ODeleteVertexStatement.java:71) at com.orientechnologies.orient.core.sql.parser.OStatement.execute(OStatement.java:72) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.command(ODatabaseDocumentEmbedded.java:647) at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.executeStatement(OServerCommandPostCommand.java:219) at com.orientechnologies.tinkerpop.command.OServerCommandPostCommandGremlin.executeStatement(OServerCommandPostCommandGremlin.java:47) at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.execute(OServerCommandPostCommand.java:104) at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommandGraph.execute(OServerCommandPostCommandGraph.java:34) at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:253) at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:811) ... 1 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 64 at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALPageChangesPortion.updateData(OWALPageChangesPortion.java:374) at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALPageChangesPortion.setIntValue(OWALPageChangesPortion.java:51) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurablePage.setIntValue(ODurablePage.java:262) at com.orientechnologies.orient.core.storage.index.sbtree.singlevalue.v3.CellBTreeSingleValueBucketV3.updateKey(CellBTreeSingleValueBucketV3.java:518) at com.orientechnologies.orient.core.storage.index.sbtree.singlevalue.v3.CellBTreeSingleValueV3.rotateNoneLeafLeftAndRemoveItem(CellBTreeSingleValueV3.java:794) at com.orientechnologies.orient.core.storage.index.sbtree.singlevalue.v3.CellBTreeSingleValueV3.deleteFromNonLeafNode(CellBTreeSingleValueV3.java:697) at com.orientechnologies.orient.core.storage.index.sbtree.singlevalue.v3.CellBTreeSingleValueV3.balanceLeafNodeAfterItemDelete(CellBTreeSingleValueV3.java:579) at com.orientechnologies.orient.core.storage.index.sbtree.singlevalue.v3.CellBTreeSingleValueV3.lambda$remove$3(CellBTreeSingleValueV3.java:532) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideComponentOperation(OAtomicOperationsManager.java:237) ... 38 more

After that I can only rester the server but the vertex are not delete

[gianluca-valentini]

UPDATE

it seems related to a class unique index

If I try to remove o rebuild it I have the same error

Removing index...
Error: com.orientechnologies.orient.core.exception.OStorageException: Exception during execution of atomic operation inside of storage Analytics
DB name="Analytics"
DB name="Analytics"

Error: com.orientechnologies.orient.core.exception.OStorageException: Exception during execution of component operation inside of storage Analytics
DB name="Analytics"

Error: java.lang.ArrayIndexOutOfBoundsException: 64

The index is present on my class Account as described below
Account.subjectFactory_userId UNIQUE ["subjectFactory","userId"] CELL_BTREE

[gianluca-valentini]

Is there something that I can do to fix this issue?

[tglman]

Hi,

fix for similar problem are in the 3.2.9 release and 3.1.x branch not yet released, will be released with 3.1.20, to fix this problem you may try to rebuild the index after upgrade, if even the index rebuild fail, the suggested solution is to export and import the data on a version with the fix.

Regards

[gianluca-valentini]

Thanks @tglman
do you know when the 3.1.20 fixed version will be released?

[gianluca-valentini]

Hi @tglman
I updated to 3.2.9 version as you suggested me.
So after that I run REBUILD INDEX *

but I have the follow error

Rebuilding index(es)...
Error: com.orientechnologies.orient.core.exception.NotEmptyComponentCanNotBeRemovedException: Recommendations.versioning : Not empty index can not be deleted. Index has 2019 records

The Class Recommendations has an index on the versioning property

but there no data are present in that property and if I run select count(*) from Recommendations I have a different value: 2059

So what I have to do to fix it?
I run REPAIR DATABASE --fix-graph too but the problem is not resolved

Is there something that I can do?
Gianluca

[tglman]

Hi,

The best way to work around a broken index is export and import the database.

fixes for avoid that the index get broken have been releases, we are working on making the remove of a broken index easier, at the end can be rebuilt from scratch.

Regards

[gianluca-valentini]

Hi @tglman
thanks a lot. I will try to export and import the database

[gianluca-valentini]

Hi @tglman
I tried to export and import the database but I have always the follow error:

Error: com.orientechnologies.orient.core.exception.OSerializationException: Error on unmarshalling JSON content for record #21:2
        DB name="Analytics2"

Error: java.lang.IllegalArgumentException: Argument '9a5335db-7a7f-4806-9eb8-18b051b3921a' is not a RecordId in form of string. Format must be: <cluster-id>:<cluster-position>

Looking the database the record #21:2 is a valid vertex

Argument '9a5335db-7a7f-4806-9eb8-18b051b3921a' is a value present in the embedded class field

where is stored a valid json (I can read/write and update it)

So I don't understand the error during import
Can you help me? What I can to do now?
Thanks
Gianluca

[gianluca-valentini]

Hi @tglman
any news about my issue?

[tglman]

Hi,

It seems that the 3.2.x series has issues with json api and import, we are addressing them, this issue are not present in 3.1.x series and the 3.1.20 has been released last week, so if is possible for you I would suggest to do the import using that version.

Regards

[gianluca-valentini]

Hi @tglman
I'm a little confused. You know that is not feasable to move always database version up and down.
I hope that coming back to 3.1.20 will fix this issue definitely

[gianluca-valentini]

Hi @tglman
I don't find the 3.1.20 docker in
https://github.com/orientechnologies/orientdb-docker/tree/d95d46e21052dbaa36c7c8cd63abde0e6826749d/release/3.1.x
Where I can find it?

[gianluca-valentini]

Hi @tglman
I see the 3.2.20 release. Does it fix the export/import bug?

[tglman]

Hi,

well is actually 3.2.10 the last release of 3.2.x series, and yes I reverted the changes that caused the problem, so it should be solved, but I could not reproduce the bug myself so I cannot confirm the fix directly.

Regards

[gianluca-valentini]

Hi @tglman
afer 2 migration and export import, finally with 3.2.10 index and export/import problems are fixed
Thanks
Gianluca

[tglman]

Hi,

Thanks, closing

[leszekgruchala]

Hey @tglman, we've got the same root exception but after a failure to remove a compound index.
Can you look at the stacktrace and let me know if the fixes you've made also cover my case? Thanks!

[tglman]

Hi,

Yes this should have been already fixed in a recent version of the 3.2, I think in 3.2.9

Regards