Skip to content
This repository has been archived by the owner on Feb 7, 2020. It is now read-only.

Out of heap memory java issue while using restore command. #42

Open
starit1977 opened this issue Sep 17, 2013 · 4 comments
Open

Out of heap memory java issue while using restore command. #42

starit1977 opened this issue Sep 17, 2013 · 4 comments

Comments

@starit1977
Copy link

Hi,

We use -

  1. elastic search version 0.90.3,
  2. Plugin version was picked up from website.
  3. Heap space size is default i.e. 256M
  4. Data size is 46MB
  5. Number of data files - Not sure how do i check this?
  6. Number of nodes - 1
  7. Index settings -
    Settings - {"test":{"settings":{"index.number_of_shards":"5","index.number_of_replicas":"1","index.version.created":"900399"}}}
    Mappings - {"test":{"export":{"properties":{"compression":{"type":"string"},"fields":{"type":"string"},"output_file":{"type":"string"}}},"device":{"properties":{"deviceId":{"type":"string"},"packages":{"properties":{"dependencies":{"properties":{"id":{"type":"string"},"version":{"type":"string"}}},"id":{"type":"string"},"source":{"type":"string"},"version":{"type":"string"}}},"properties":{"properties":{"pin":{"properties":{"source":{"type":"string"},"value":{"type":"string"}}}}},"supportedAPIs":{"properties":{"id":{"type":"string"},"source":{"type":"string"},"type":{"type":"string"},"version":{"type":"string"}}},"ts":{"type":"long"},"vendorId":{"type":"string"}}}}}

The index name is test and it has around 45 MB worth of data.
The dump worked perfectly fine. However, when i use restore it gave me out of heap error.

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/data/elasticsearch/nodes/0/dump$
local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/data/elasticsearch/nodes/0/dump$
local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/data/elasticsearch/nodes/0/dump$ curl -X POST 'http://localhost:9200/test/_restore' [2013-09-17 17:16:50,035][INFO ][cluster.metadata ] [Eric] [test] creating index, cause [api], shards [5]/[1], mappings []
[2013-09-17 17:16:50,076][DEBUG][index.gateway ] [Eric] [test][0] starting recovery from local ...
[2013-09-17 17:16:50,082][DEBUG][index.gateway ] [Eric] [test][1] starting recovery from local ...
[2013-09-17 17:16:50,084][DEBUG][index.gateway ] [Eric] [test][0] recovery completed from local, took [8ms]
index : files [0] with total_size [0b], took[0s]
: recovered_files [0] with total_size [0b]
: reusing_files [0] with total_size [0b]
start : took [8ms], check_index [0s]
translog : number_of_operations [0], took [0s]
[2013-09-17 17:16:50,092][DEBUG][index.gateway ] [Eric] [test][1] recovery completed from local, took [10ms]
index : files [0] with total_size [0b], took[0s]
: recovered_files [0] with total_size [0b]
: reusing_files [0] with total_size [0b]
start : took [10ms], check_index [0s]
translog : number_of_operations [0], took [0s]
[2013-09-17 17:16:50,101][DEBUG][index.gateway ] [Eric] [test][2] starting recovery from local ...
[2013-09-17 17:16:50,107][INFO ][cluster.metadata ] [Eric] [test] create_mapping [export]
[2013-09-17 17:16:50,108][DEBUG][index.gateway ] [Eric] [test][2] recovery completed from local, took [7ms]
index : files [0] with total_size [0b], took[1ms]
: recovered_files [0] with total_size [0b]
: reusing_files [0] with total_size [0b]
start : took [6ms], check_index [0s]
translog : number_of_operations [0], took [0s]
[2013-09-17 17:16:50,130][DEBUG][index.gateway ] [Eric] [test][3] starting recovery from local ...
[2013-09-17 17:16:50,138][DEBUG][index.gateway ] [Eric] [test][4] starting recovery from local ...
[2013-09-17 17:16:50,139][DEBUG][index.gateway ] [Eric] [test][3] recovery completed from local, took [10ms]
index : files [0] with total_size [0b], took[0s]
: recovered_files [0] with total_size [0b]
: reusing_files [0] with total_size [0b]
start : took [9ms], check_index [0s]
translog : number_of_operations [0], took [0s]
[2013-09-17 17:16:50,147][DEBUG][index.gateway ] [Eric] [test][4] recovery completed from local, took [9ms]
index : files [0] with total_size [0b], took[0s]
: recovered_files [0] with total_size [0b]
: reusing_files [0] with total_size [0b]
start : took [8ms], check_index [0s]
translog : number_of_operations [0], took [0s]
[2013-09-17 17:16:50,163][INFO ][cluster.metadata ] [Eric] [test] create_mapping [device]
[2013-09-17 17:17:00,770][WARN ][index.engine.robin ] [Eric] [test][2] failed engine
java.lang.OutOfMemoryError: Java heap space
[2013-09-17 17:17:05,837][DEBUG][action.bulk ] [Eric] [test][2] failed to execute bulk item (index) index {[test][device][1336553], source[{"data......................"}]}
org.elasticsearch.index.engine.IndexFailedEngineException: [test][2] Index failed for [device#1336553]
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:503)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:341)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:397)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:521)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.lucene.store.AlreadyClosedException: this ReferenceManager is closed
at org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:97)
at org.elasticsearch.index.engine.robin.RobinEngine.searcher(RobinEngine.java:739)
at org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java:1319)
at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:516)
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:490)
... 8 more
[2013-09-17 17:17:06,027][DEBUG][action.bulk ] [Eric] [test][2] failed to execute bulk item (index) index {[test][device][1156463], source[{"data................."}]}
org.elasticsearch.index.engine.IndexFailedEngineException: [test][2] Index failed for [device#1156463]
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:503)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:341)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:397)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:521)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.lucene.store.AlreadyClosedException: this ReferenceManager is closed
at org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:97)
at org.elasticsearch.index.engine.robin.RobinEngine.searcher(RobinEngine.java:739)
at org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java:1319)
at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:516)
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:490)
... 8 more
[2013-09-17 17:17:06,039][DEBUG][action.bulk ] [Eric] [test][2], node[vZI558XRT1i1kGPb4OXXnA], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.bulk.BulkShardRequest@5433050b]
java.lang.NullPointerException
[2013-09-17 17:17:06,038][DEBUG][action.bulk ] [Eric] [test][2], node[vZI558XRT1i1kGPb4OXXnA], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.bulk.BulkShardRequest@1fede67b]
java.lang.NullPointerException
[2013-09-17 17:17:06,044][WARN ][index.translog ] [Eric] [test][2] failed to flush shard on translog threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [test][2] Flush failed
at org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:895)
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:502)
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:186)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.OutOfMemoryError: Java heap space
[2013-09-17 17:17:06,045][DEBUG][action.bulk ] [Eric] [test][2] failed to execute bulk item (index) index {[test][device][1208327], source[{"data...."}]}
org.elasticsearch.index.engine.IndexFailedEngineException: [test][2] Index failed for [device#1208327]
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:498)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:341)

Any input much appreciated.

Thanks
Pranav.
@starit1977
Copy link
Author

Note - If, I increase the Heap size more than the default, then it works fine. But i don't know how much Heap size should be set ideally to make this work? The command that gave error used the Default heap size i.e. 256 MB.
We are planning to use this plugin for backup/restore/export/import like 50GB of data.

Any input on the above error, much appreciated.

Thanks
Pranav.

@mfussenegger
Copy link
Member

This is probably related with #35 and should be fixed with 0.90.4

@starit1977
Copy link
Author

I'm installing elastic search 0.90.5 version and see if i can get it working.

Between - i'm not a java guy, is there any simple set of steps that can help me install this plugin. I felt it is complicated, since again i'm not a java guy.

Thanks again for quick reply.

@apatrida
Copy link

for 0.90.5 see the pull request for that version, there are changes you need in the code to safely use all of the plugin

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mfussenegger @apatrida @starit1977