You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Jun 1, 2021. It is now read-only.
Currently we're getting OutOfMemoryErrors in a node that's just replicating events, running with leveldb. Because the node is only replicating events we assumed that it does not need much memory, therefore it's running with 1G max heap.
The heap dump analysis showed that eventuate's LeveldbNumericIdentifierStore consumes ~1.3GB RAM, which is allocated by the internal idMap (with ~6.5M entries):
AFAICS the idMap stores aggregate ids, right?
For now we'll increase the max heap size of the java process (which we could also keep as solution, assuming that the string ids are aggregate ids).
It might still be considered to reduce the memory requirements of the leveldb event log.
Because the LeveldbNumericIdentifierStore stores Int -> String (numeric id to string id) in leveldb, it's currently not possible to resolve a string id to its numeric id via leveldb (in def numericId(id: String): Int). I.e. we cannot simply get rid of the idMap. If we don't want to also store the String -> Int mapping in leveldb we'd have to use a more memory efficient Map implementation for this use case.
WDYT?
The text was updated successfully, but these errors were encountered:
We should consider the Int->String storage design as obsolete (it has historical reasons which do not apply any more) and directly store the String->Int mapping in LevelDB (as we don't need a key ordering here and hence don't need keys of equal length). For backward compatibility this requires DB migration.
Currently we're getting
OutOfMemoryError
s in a node that's just replicating events, running with leveldb. Because the node is only replicating events we assumed that it does not need much memory, therefore it's running with 1G max heap.The heap dump analysis showed that eventuate's LeveldbNumericIdentifierStore consumes ~1.3GB RAM, which is allocated by the internal idMap (with ~6.5M entries):
AFAICS the
idMap
stores aggregate ids, right?For now we'll increase the max heap size of the java process (which we could also keep as solution, assuming that the string ids are aggregate ids).
It might still be considered to reduce the memory requirements of the leveldb event log.
Because the
LeveldbNumericIdentifierStore
storesInt -> String
(numeric id to string id) in leveldb, it's currently not possible to resolve a string id to its numeric id via leveldb (indef numericId(id: String): Int
). I.e. we cannot simply get rid of theidMap
. If we don't want to also store theString -> Int
mapping in leveldb we'd have to use a more memory efficientMap
implementation for this use case.WDYT?
The text was updated successfully, but these errors were encountered: