[Feature request] Add periodic commit module

[double-di]

Hi!
We are in the middle of switching from neo4j to memgraph.
We would love to see
https://neo4j.com/labs/apoc/4.1/overview/apoc.periodic/apoc.periodic.commit/
in this module, since a lot of our queries are using it.

Thanks a lot!

[karmenrabar]

Hello @double-di ! Thank you for your interest in the periodic commit module. To better understand the situation, could you provide more details on the specific use case or scenarios where this module is being considered? Do you have a workaround for the moment? Your input will help us evaluate the impact and prioritization of the periodic commit module.

[gitbuda]

Related issue memgraph/memgraph#1182

[double-di]

@karmenrabar Hi! Let me try to explain our usecase. Our database consist of 5B nodes & 5B relations. 100 M nodes are marked as "Starred". For any node we are trying to answer the question "Is it connected to a starred node with 16 hops or not? If it is connected - show any shortest path". Since the amount of nodes & relations is big it is not possible to answer this question in realtime by executing BFS directly. So for every "Starred" node we are precalculating (connecting with "CLOSE_TO_STARRED" relations to all the connected nodes) so we can answer this question fast. This process is very long and is implemented incrementally using flags "should_calculated_stared" on the nodes that should be calculated and apoc.periodic.commit to calculate a batch of nodes -> commit -> proceed

[antejavor]

Hi @double-di, thanks a lot for providing an in-depth explanation of this feature request 🚀 . Based on your explanation, I gave this feature request a bit higher priority.

How is your Memgraph experience overall?

Just for info, we are hosting Office hours calls. If you need any help from the Memgraph team in your journey or provide feedback, it could be worth while to jump on a short engineering-driven call.

[double-di]

Thanks a lot for your support @antejavor . We are currently on early adoption stage

[katarinasupe]

Hi @double-di, just to keep you posted - you can track progress for periodic commit on Memgraph repo, since it will be a part of Cypher. How is your project with Memgraph going? At what stage are you now?