Database import fails #2985

joha78 · 2014-10-27T16:32:29Z

Dear OrientDB-Team,
I am fighting with a 4.1 GB large database export. I tried version 1.7.7 and now 1.7.9 with default configuration (except that I added -Xmx2048m to server.sh).

My export contains about 20,000 small PDF-Files. As I import it (using console.sh) into a fresh created database it crashes the database server JVM. What could I try?

With best wishes from Germany
Jörg

At the side of console.sh I get this exception:

Error importing record #3:-1. Source line 0, column 709695529
Error on database import happened just before line 0, column 709695530
com.orientechnologies.orient.core.exception.OStorageException: Error on create record in cluster: 3
    at com.orientechnologies.orient.client.remote.OStorageRemote.handleException(OStorageRemote.java:1593)
    at com.orientechnologies.orient.client.remote.OStorageRemote.createRecord(OStorageRemote.java:390)
    at com.orientechnologies.orient.client.remote.OStorageRemoteThread.createRecord(OStorageRemoteThread.java:236)
    at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.save(ODatabaseRaw.java:271)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:1136)
    at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:84)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:322)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:40)
    at com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:335)
    at com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:326)
    at com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:42)
    at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecord(ODatabaseImport.java:1216)
    at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1112)
    at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:421)
    at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:1572)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:365)
    at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:230)
    at com.orientechnologies.common.console.OConsoleApplication.executeBatch(OConsoleApplication.java:189)
    at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:141)
    at com.orientechnologies.orient.graph.console.OGremlinConsole.main(OGremlinConsole.java:55)
Caused by: java.io.IOException: Channel is closed
    at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:181)
    at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:160)
    at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1843)
    at com.orientechnologies.orient.client.remote.OStorageRemote.createRecord(OStorageRemote.java:338)
    ... 22 more

Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'workplace' from file: /home/atsd/orientdb-export.json.gz

Error: com.orientechnologies.orient.core.exception.OStorageException: Error on create record in cluster: 3

Error: java.io.IOException: Channel is closed

Error on retrieving the connection from pool: 127.0.0.1:2424/workplace
Error on connecting to 127.0.0.1:2424/workplace
-> com.orientechnologies.orient.client.remote.ORemoteConnectionManager.createNetworkConnection(ORemoteConnectionManager.java:210)
-> com.orientechnologies.orient.client.remote.ORemoteConnectionManager$1.createNewResource(ORemoteConnectionManager.java:74)
-> com.orientechnologies.orient.client.remote.ORemoteConnectionManager$1.createNewResource(ORemoteConnectionManager.java:71)
-> com.orientechnologies.common.concur.resource.OResourcePool.getResource(OResourcePool.java:82)
-> com.orientechnologies.orient.client.remote.ORemoteConnectionManager.acquire(ORemoteConnectionManager.java:94)
-> com.orientechnologies.orient.client.remote.OStorageRemote.getAvailableNetwork(OStorageRemote.java:1815)
-> com.orientechnologies.orient.client.remote.OStorageRemote.beginRequest(OStorageRemote.java:1792)
-> com.orientechnologies.orient.client.remote.OStorageRemote.close(OStorageRemote.java:258)
-> com.orientechnologies.orient.client.remote.OStorageRemoteThread.close(OStorageRemoteThread.java:108)
-> com.orientechnologies.orient.core.Orient.closeAllStorages(Orient.java:237)
-> com.orientechnologies.orient.core.Orient.shutdown(Orient.java:184)
-> com.orientechnologies.orient.core.OrientShutdownHook.run(OrientShutdownHook.java:28)
Verbindungsaufbau abgelehnt
-> com.orientechnologies.orient.client.remote.ORemoteConnectionManager.createNetworkConnection(ORemoteConnectionManager.java:210)
-> com.orientechnologies.orient.client.remote.ORemoteConnectionManager$1.createNewResource(ORemoteConnectionManager.java:74)
-> com.orientechnologies.orient.client.remote.ORemoteConnectionManager$1.createNewResource(ORemoteConnectionManager.java:71)
-> com.orientechnologies.common.concur.resource.OResourcePool.getResource(OResourcePool.java:82)
-> com.orientechnologies.orient.client.remote.ORemoteConnectionManager.acquire(ORemoteConnectionManager.java:94)
-> com.orientechnologies.orient.client.remote.OStorageRemote.getAvailableNetwork(OStorageRemote.java:1815)
-> com.orientechnologies.orient.client.remote.OStorageRemote.beginRequest(OStorageRemote.java:1792)
-> com.orientechnologies.orient.client.remote.OStorageRemote.close(OStorageRemote.java:258)
-> com.orientechnologies.orient.client.remote.OStorageRemoteThread.close(OStorageRemoteThread.java:108)
-> com.orientechnologies.orient.core.Orient.closeAllStorages(Orient.java:237)
-> com.orientechnologies.orient.core.Orient.shutdown(Orient.java:184)
-> com.orientechnologies.orient.core.OrientShutdownHook.run(OrientShutdownHook.java:28)

At the server side I get:

OrientDB.log **************
java.lang.OutOfMemoryError: Metaspace
Dumping heap to java_pid29955.hprof ...
Heap dump file created [12783002 bytes in 0,473 secs]
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 205728 bytes for Chunk::new
# An error report file with more information is saved as:
# /home/atsd/orientdb-community-1.7.9/bin/hs_err_pid29955.log
#
# Compiler replay data is saved as:
# /home/atsd/orientdb-community-1.7.9/bin/replay_pid29955.log
OrientDB.log **************

OrientDB.err **************
 [ONetworkProtocolBinary]Exception in thread "OrientDB <- BinaryClient (/127.0.0.1:44937)" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "OrientDB <- BinaryClient (/127.0.0.1:44937)"
OrientDB.err **************

orient-server.log is empty.

The text was updated successfully, but these errors were encountered:

lvca · 2014-10-27T16:46:55Z

Have you already tried importing it without server but as embedded using plocal URL?

joha78 · 2014-10-27T17:31:51Z

Good idea! I tried this now. Result is the same :-(

Strange, it seems that as bigger I make -Xmx, the less records I get into the database. With default JVM memory config default.pcl grew until 1.4 GB, with -Xmx2048m it failed earlier at about 766MB.

Any idea is appreciated!

*********console.sh output

Importing records...

Importing records into cluster 'internal' (id=0) = 2 records
Importing records into cluster 'default' (id=3)Error importing record Add trigger feature [moved] #3:-1. Source line 0, column 934746013
Error on database import happened just before line 0, column 934746014
com.orientechnologies.orient.core.exception.OStorageException: Error on creating record in cluster: com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster@15189ac
at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.createRecord(OLocalPaginatedStorage.java:690)
at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.save(ODatabaseRaw.java:271)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:1136)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:84)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:322)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:40)
at com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:335)
at com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:326)
at com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:42)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecord(ODatabaseImport.java:1216)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1112)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:421)
at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:1572)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:365)
at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:230)
at com.orientechnologies.common.console.OConsoleApplication.executeBatch(OConsoleApplication.java:189)
at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:141)
at com.orientechnologies.orient.graph.console.OGremlinConsole.main(OGremlinConsole.java:55)
Caused by: com.orientechnologies.orient.core.exception.OStorageException
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.createRecord(OPaginatedCluster.java:556)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.createRecord(OLocalPaginatedStorage.java:673)
... 21 more
Caused by: java.lang.OutOfMemoryError
at com.orientechnologies.nio.OJNADirectMemory.allocate(OJNADirectMemory.java:37)
at com.orientechnologies.common.directmemory.ODirectMemoryPointer.(ODirectMemoryPointer.java:50)
at com.orientechnologies.orient.core.index.hashindex.local.cache.OWOWCache.cacheFileContent(OWOWCache.java:1077)
at com.orientechnologies.orient.core.index.hashindex.local.cache.OWOWCache.load(OWOWCache.java:617)
at com.orientechnologies.orient.core.index.hashindex.local.cache.OReadWriteDiskCache.updateCache(OReadWriteDiskCache.java:476)
at com.orientechnologies.orient.core.index.hashindex.local.cache.OReadWriteDiskCache.load(OReadWriteDiskCache.java:200)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.addEntry(OPaginatedCluster.java:1037)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.createRecord(OPaginatedCluster.java:510)
... 22 more

Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'workplace' from file: /home/atsd/orientdb-export.json.gz

Error: com.orientechnologies.orient.core.exception.OStorageException: Error on creating record in cluster: com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster@15189ac

Error: com.orientechnologies.orient.core.exception.OStorageException

Error: java.lang.OutOfMemoryError

lvca · 2014-10-27T17:35:16Z

How much memory PHYSICAL do you have on your PC and free before to start importer?

joha78 · 2014-10-27T18:02:55Z

Free is 1300 MB, and there are few GB virtual memory available.

lvca · 2014-10-27T18:28:01Z

Ok,
set the maxheap to 512M and storage.diskCache.bufferSize to 600Mb:

java ... -Xmx512M -Dstorage.diskCache.bufferSize=600 ...

joha78 · 2014-10-27T19:26:57Z

This worked perfectly for me. Thank you very much!

joha78 closed this as completed Oct 27, 2014

lvca added wontfix invalid labels Oct 27, 2014

lvca self-assigned this Oct 27, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Database import fails #2985

Database import fails #2985

joha78 commented Oct 27, 2014

lvca commented Oct 27, 2014

joha78 commented Oct 27, 2014

lvca commented Oct 27, 2014

joha78 commented Oct 27, 2014

lvca commented Oct 27, 2014

joha78 commented Oct 27, 2014

Database import fails #2985

Database import fails #2985

Comments

joha78 commented Oct 27, 2014

lvca commented Oct 27, 2014

joha78 commented Oct 27, 2014

lvca commented Oct 27, 2014

joha78 commented Oct 27, 2014

lvca commented Oct 27, 2014

joha78 commented Oct 27, 2014