Memory usage problem

[ag-michael]

Hi,

I am having issues with neo4j using too much memory as a result of goodhound execution, I have not been able to get goodhound to finish running at all because of this, even with a low '-r' parameter.

I have set the max heap size to 28GB, even then it fails with the error below, I cannot tell what specific query is causing this. Can the script or query be optimized in anyway to avoid this problem?

DEBUG:Adding connection pool for profile ConnectionProfile('bolt://localhost:7687')
DEBUG:Trying to acquire connection from pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:[#0000] C: (Dialing <localhost:7687>)
DEBUG:[#EB39] S: (Accepted)
DEBUG:[#EB39] C: <BOLT>
DEBUG:[#EB39] C: <PROTOCOL> 4.3~4.0 | 4.0 | 3.0 | 2.0
DEBUG:[#EB39] S: <PROTOCOL> 4.3
DEBUG:[#EB39] C: HELLO {'user_agent': 'py2neo/2021.2.3 Python/3.11.1-final-0 (win32)', 'scheme': 'basic', 'principal': 'neo4j', 'credentials': '*******'}
DEBUG:[#EB39] C: (Sent 133 bytes)
DEBUG:[#EB39] S: SUCCESS {'server': 'Neo4j/4.4.12', 'connection_id': 'bolt-138', 'hints': {}}
DEBUG:Releasing connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> from thread <_MainThread(MainThread, started 3432)>
DEBUG:Connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> does not belong to pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:Connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> acquired by thread <_MainThread(MainThread, started 3432)>
DEBUG:Releasing connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> from thread <_MainThread(MainThread, started 3432)>
Warming up database
DEBUG:Attempting to acquire read-write connection to default database
DEBUG:Using connection pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:Trying to acquire connection from pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:Connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> acquired by thread <_MainThread(MainThread, started 3432)>
DEBUG:[#EB39] C: RUN 'MATCH (n) OPTIONAL MATCH (n)-[r]->() RETURN count(n.name) + count(r.isacl)' {} {}
DEBUG:[#EB39] C: PULL {'n': -1, 'qid': -1}
DEBUG:[#EB39] C: (Sent 99 bytes)
DEBUG:[#EB39] S: SUCCESS {'t_first': 15, 'fields': ['count(n.name) + count(r.isacl)']}
DEBUG:[#EB39] S: RECORD * 1
DEBUG:[#EB39] S: SUCCESS {'bookmark': 'FB:', 'type': 'r', 't_last': 31939, 'db': 'neo4j'}
DEBUG:Releasing connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> from thread <_MainThread(MainThread, started 3432)>
INFO:Setting cost.
DEBUG:Attempting to acquire read-write connection to default database
DEBUG:Using connection pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:Trying to acquire connection from pool <py2neo.client.ConnectionPool object at 0x000001CEE1CA48D0>
DEBUG:Connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> acquired by thread <_MainThread(MainThread, started 3432)>
DEBUG:[#EB39] C: RUN 'MATCH (n)-[r:MemberOf]->(m:Group) SET r.cost = 0' {} {}
DEBUG:[#EB39] C: PULL {'n': -1, 'qid': -1}
DEBUG:[#EB39] C: (Sent 73 bytes)
DEBUG:[#EB39] S: SUCCESS {'t_first': 12547, 'fields': []}
DEBUG:[#EB39] S: FAILURE {'code': 'Ne
o.TransientError.General.OutOfMemoryError', 'message': "There is not enough memory to perform the current task. Please try increasing 'dbms.memory.heap.max_size' in the neo4j configuration (normally in 'conf/neo4j.conf' or, if you are using Neo4j Desktop, found through the user interface) or if you are running an embedded installation increase the heap by using '-Xmx' command line flag, and then restart the database."}
DEBUG:[#EB39] C: RESET
DEBUG:[#EB39] C: (Sent 6 bytes)
DEBUG:[#EB39] S: SUCCESS {}
DEBUG:Releasing connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> from thread <_MainThread(MainThread, started 3432)>
DEBUG:Releasing connection <py2neo.client.bolt.Bolt4x3 object at 0x000001CEE2085510> from thread <_MainThread(MainThread, started 3432)>
WARNING:Error setting cost!

[idnahacks]

Hi, apologies for the very late response. I haven't been able to actively maintain Goodhound since April 2022.

When I was running this regularly on a reasonably large AD dataset I didn't see memory usage like that. You could try running some of the queries inside the code directly in the neo4j interface and see if you get similar issues in there.

It could be something funky in your dataset that is causing a loop, such as a looping memberof. I've tried to compensate for that in the code, but did not have enough real world datasets to discover every use case.