Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

memory leak during full database import #4461

Closed
jingjingbic opened this issue Jun 25, 2015 · 8 comments
Closed

memory leak during full database import #4461

jingjingbic opened this issue Jun 25, 2015 · 8 comments
Assignees
@andrii0lomakin
Labels
question
Milestone
2.1 GA

Comments

@jingjingbic
Copy link

Hi orientdb team,

We see this error in both 2.0.10 and 2.0.11, in linux. This is that we did:

[jing@ndexws1 bin]$ ./console.sh

OrientDB console v.2.0.11 (build UNKNOWN@r; 2015-06-19 15:38:17+0000) www.orientechnologies.com
Type 'help' to display all the supported commands.
Installing extensions for GREMLIN language v.2.6.0

orientdb> create database plocal:/opt/ndex/orientdb-community-2.0.11/databases/ndex admin admin

Creating database [plocal:/opt/ndex/orientdb-community-2.0.11/databases/ndex] using the storage type [plocal]...
Database created successfully.

Current database is: plocal:/opt/ndex/orientdb-community-2.0.11/databases/ndex
orientdb {db=ndex}> import database /home/chenjing/db_2015-06-24.export.gz

Importing database database /home/chenjing/db_2015-06-24.export.gz...
Started import of database 'plocal:/opt/ndex/orientdb-community-2.0.11/databases/ndex' from /home/chenjing/db_2015-06-24.export.gz...
Non merge mode (-merge=false): removing all default non security classes

  • Class E was removed.
    ......

After a while, in the 2.0.10 machine, we got this error message:

  • Imported 34,218 records into clusters: [baseterm]. Total records imported so far: 10,531,246 (6,843.60/sec)Error during WAL background flush
    java.lang.OutOfMemoryError: Java heap space
    at com.orientechnologies.common.directmemory.OUnsafeMemoryJava7.get(OUnsafeMemoryJava7.java:30)
    at com.orientechnologies.common.directmemory.ODirectMemoryPointer.get(ODirectMemoryPointer.java:79)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog$LogSegment$FlushTask.commit(ODiskWriteAheadLog.java:192)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog$LogSegment$FlushTask.run(ODiskWriteAheadLog.java:129)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    Error during WAL background flush
    java.lang.OutOfMemoryError: Java heap space
    at com.orientechnologies.common.directmemory.OUnsafeMemoryJava7.get(OUnsafeMemoryJava7.java:30)
    at com.orientechnologies.common.directmemory.ODirectMemoryPointer.get(ODirectMemoryPointer.java:79)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog$LogSegment$FlushTask.commit(ODiskWriteAheadLog.java:192)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog$LogSegment$FlushTask.run(ODiskWriteAheadLog.java:129)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    Error during WAL background flush
    java.lang.OutOfMemoryError: Java heap space
    at com.orientechnologies.common.directmemory.OUnsafeMemoryJava7.get(OUnsafeMemoryJava7.java:30)
    at com.orientechnologies.common.directmemory.ODirectMemoryPointer.get(ODirectMemoryPointer.java:79)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog$LogSegment$FlushTask.commit(ODiskWriteAheadLog.java:192)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog$LogSegment$FlushTask.run(ODiskWriteAheadLog.java:129)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    Error during WAL background flush
    java.lang.OutOfMemoryError: Java heap space
    Error during WAL background flush
    java.lang.OutOfMemoryError: Java heap space
    Error during WAL background flush
    java.lang.OutOfMemoryError: Java heap space
    Error during WAL background flush
    java.lang.OutOfMemoryError: Java heap space
    Error during WAL background flush
    java.lang.OutOfMemoryError: Java heap space
    Error during WAL background flush
    java.lang.OutOfMemoryError: Java heap space
    {db=ndex} Error on creating record in cluster: com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster@6f03d792
    com.orientechnologies.orient.core.index.sbtree.local.OSBTreeException: Error during index update with key Auto discovery of databases if they are placed under $ORIENT_HOME/databases [moved] #23:5306397 and value 1
    at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiLocal.put(OSBTreeBonsaiLocal.java:319)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.ORidBagUpdateSerializationOperation.execute(ORidBagUpdateSerializationOperation.java:63)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.ORecordSerializationContext.executeOperations(ORecordSerializationContext.java:99)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doCreateRecord(OAbstractPaginatedStorage.java:1502)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.createRecord(OAbstractPaginatedStorage.java:654)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1745)
    at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:102)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2306)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:118)
    at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1734)
    at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1725)
    at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:98)
    at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecord(ODatabaseImport.java:1291)
    at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1187)
    at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:425)
    at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:1534)
    at com.orientechnologies.orient.graph.console.OGremlinConsole.importDatabase(OGremlinConsole.java:124)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:383)
    at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:248)
    at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:137)
    at com.orientechnologies.orient.graph.console.OGremlinConsole.main(OGremlinConsole.java:62)
    Caused by: java.lang.OutOfMemoryError: Java heap space

2015-06-25 01:40:52:007 SEVERE {db=ndex} Error importing record #20:4064633. Source line 0, column -1530632691 [ODatabaseImport]Error on database import happened just before line 0, column -1530632690
com.orientechnologies.orient.core.exception.ODatabaseException: Error on saving record #20:4064633
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1789)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:102)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2306)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:118)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1734)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1725)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:98)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecord(ODatabaseImport.java:1291)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1187)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:425)
at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:1534)
at com.orientechnologies.orient.graph.console.OGremlinConsole.importDatabase(OGremlinConsole.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:383)
at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:248)
at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:137)
at com.orientechnologies.orient.graph.console.OGremlinConsole.main(OGremlinConsole.java:62)
Caused by: java.lang.NullPointerException
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1747)
... 19 more

Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'ndex' from file: /opt/ndex/orientdb/databases/db_2015-06-24.export.gz

Error: com.orientechnologies.orient.core.exception.ODatabaseException: Error on saving record #20:4064633

Error: java.lang.NullPointerException

orientdb {db=ndex}>

On the 2.0.11 machine, we got this error message:

  • Imported 32,390 records into clusters: [baseterm]. Total records imported so far: 10,480,050 (6,478.00/sec)
  • Imported 33,556 records into clusters: [baseterm]. Total records imported so far: 10,513,606 (6,711.20/sec){db=ndex} Error on creating record in cluster: com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster@26f1ad8e
    java.lang.OutOfMemoryError: Java heap space
    at java.util.ArrayDeque.(ArrayDeque.java:188)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALChangesTree.applyChanges(OWALChangesTree.java:248)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALChangesTree.getBinaryValue(OWALChangesTree.java:58)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALChangesTree$PointerWrapper.get(OWALChangesTree.java:792)
    at com.orientechnologies.orient.core.serialization.serializer.binary.impl.OLinkSerializer.deserializeFromDirectMemoryObject(OLinkSerializer.java:122)
    at com.orientechnologies.orient.core.serialization.serializer.binary.impl.OLinkSerializer.deserializeFromDirectMemoryObject(OLinkSerializer.java:43)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurablePage.deserializeFromDirectMemory(ODurablePage.java:129)
    at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiBucket.getKey(OSBTreeBonsaiBucket.java:285)
    at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiBucket.find(OSBTreeBonsaiBucket.java:203)
    at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiLocal.findBucket(OSBTreeBonsaiLocal.java:1184)
    at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiLocal.get(OSBTreeBonsaiLocal.java:188)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.ORidBagUpdateSerializationOperation.execute(ORidBagUpdateSerializationOperation.java:57)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.ORecordSerializationContext.executeOperations(ORecordSerializationContext.java:99)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doCreateRecord(OAbstractPaginatedStorage.java:1502)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.createRecord(OAbstractPaginatedStorage.java:654)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1748)
    at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:102)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2309)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:118)
    at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1727)
    at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1718)
    at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:76)
    at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecord(ODatabaseImport.java:1291)
    at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1187)
    at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:425)
    at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:1566)
    at com.orientechnologies.orient.graph.console.OGremlinConsole.importDatabase(OGremlinConsole.java:124)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:383)

2015-06-25 02:28:23:214 SEVERE {db=ndex} Error importing record #20:4064633. Source line 0, column -1530632691 [ODatabaseImport]Error on database import happened just before line 0, column -1530632690
com.orientechnologies.orient.core.exception.ODatabaseException: Error on saving record #20:4064633
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1792)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:102)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2309)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:118)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1727)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1718)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:76)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecord(ODatabaseImport.java:1291)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1187)
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:425)
at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:1566)
at com.orientechnologies.orient.graph.console.OGremlinConsole.importDatabase(OGremlinConsole.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:383)
at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:248)
at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:137)
at com.orientechnologies.orient.graph.console.OGremlinConsole.main(OGremlinConsole.java:62)
Caused by: java.lang.NullPointerException
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1750)
... 19 more

Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'ndex' from file: /home/chenjing/db_2015-06-24.export.gz

Error: com.orientechnologies.orient.core.exception.ODatabaseException: Error on saving record #20:4064633

Error: java.lang.NullPointerException

orientdb {db=ndex}>

This is a very critical problem for us because we rely on the export/import command to back restore our production database, we trie this on a machine with 60G machine but still get outOfMemory error and couldn't restore the database from a back.

If it helps, I can provide the exported db file for you to reproduce this error.
@andrii0lomakin
Copy link
Member

Could you provide memory dump which was created during OOM exception.

@jingjingbic
Copy link
Author

Andrey,

I have send you the link of the db file to your gmail account. and you should be able to download the db dump and reproduce the error.

@jingjingbic
Copy link
Author

Hi Andrey,

I tried the db import again in 2.0.12 and it still failed. I have sent the memory dump to you via email.
Looks like a HashMap in this object is using a log memories.
com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperation

Class Name | Shallow Heap | Retained Heap | Percentage

com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperation @ 0xf1567200| 48 | 203,485,624 | 41.33%

@lvca lvca added this to the 2.0.13 milestone Jul 10, 2015
@lvca lvca modified the milestones: 2.0.13, 2.1 GA Jul 22, 2015
@andrii0lomakin
Copy link
Member

Hi,
I looked on your heap dump do you have only 512 MB of RAM ?

@andrii0lomakin
Copy link
Member

Oh sorry that is console settings I will try to reproduce it, the most preferable would be if we would able to import database in small amount of memory but if it would not be possible I will try to import it with more ram provided by console app.

@andrii0lomakin
Copy link
Member

Hi,
Could you use following command line parameters in console to fix given issue -Xmx2048m -Dstorage.useWAL=false (but only during database import) . Your problem is that you have vertexes with huge amount of edges which causes big size of transaction in memory , In 2.2 we are going to use data structure which consumes less memory, so you will not need to use those settings.

@lvca lvca added the question label Jul 30, 2015
@jingjingbic
Copy link
Author

Thanks Adrey. Does the -Xmx2048m heap setting correlate to the size of the database or to the maximum size of edge counts on vertexes?

@andrii0lomakin
Copy link
Member

It correlates with maximum counts of edges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andrii0lomakin andrii0lomakin

Labels
question
Milestone
2.1 GA
Development

No branches or pull requests
3 participants
@andrii0lomakin @lvca @jingjingbic