Skip to content
This repository has been archived by the owner on Jun 1, 2021. It is now read-only.

Memory consumption of LeveldbNumericIdentifierStore #402

Open
magro opened this issue Jan 12, 2018 · 2 comments
Open

Memory consumption of LeveldbNumericIdentifierStore #402

magro opened this issue Jan 12, 2018 · 2 comments

Comments

@magro
Copy link
Contributor

magro commented Jan 12, 2018

Currently we're getting OutOfMemoryErrors in a node that's just replicating events, running with leveldb. Because the node is only replicating events we assumed that it does not need much memory, therefore it's running with 1G max heap.

The heap dump analysis showed that eventuate's LeveldbNumericIdentifierStore consumes ~1.3GB RAM, which is allocated by the internal idMap (with ~6.5M entries):
sshot_sync001_heapdump_idstore

AFAICS the idMap stores aggregate ids, right?

For now we'll increase the max heap size of the java process (which we could also keep as solution, assuming that the string ids are aggregate ids).
It might still be considered to reduce the memory requirements of the leveldb event log.

Because the LeveldbNumericIdentifierStore stores Int -> String (numeric id to string id) in leveldb, it's currently not possible to resolve a string id to its numeric id via leveldb (in def numericId(id: String): Int). I.e. we cannot simply get rid of the idMap. If we don't want to also store the String -> Int mapping in leveldb we'd have to use a more memory efficient Map implementation for this use case.

WDYT?

@krasserm
Copy link
Contributor

We should consider the Int->String storage design as obsolete (it has historical reasons which do not apply any more) and directly store the String->Int mapping in LevelDB (as we don't need a key ordering here and hence don't need keys of equal length). For backward compatibility this requires DB migration.

@magro
Copy link
Contributor Author

magro commented Jan 20, 2018

Ok, thanks for the hint!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants