From c88bd4d0d335307eefb48687bcbfa68bcdf6163d Mon Sep 17 00:00:00 2001 From: Josip Mrden Date: Wed, 22 May 2024 12:24:48 +0200 Subject: [PATCH 1/4] Peak memory observation in the system --- pages/database-management/server-stats.md | 1 + 1 file changed, 1 insertion(+) diff --git a/pages/database-management/server-stats.md b/pages/database-management/server-stats.md index 50d445ae8..3f9d42fee 100644 --- a/pages/database-management/server-stats.md +++ b/pages/database-management/server-stats.md @@ -33,6 +33,7 @@ The result will contain the following fields: | edge_count | The number of stored relationships (edges). | | average_degree | The average number of relationships of a single node. | | memory_res | The non-swapped physical RAM memory a task has used, reported by the OS (in B, KiB, MiB, GiB or TiB). | +| peak_memory_res | Peak RAM memory usage in the system during the whole run. | | disk_usage | The amount of disk space used by the data directory (in B, KiB, MiB, GiB or TiB). | | memory_tracked | The amount of RAM allocated in the system and tracked by Memgraph (in B, KiB, MiB, GiB or TiB).
For more info, check out [memory control](/fundamentals/storage-memory-usage). | | vm_max_map_count | The number of memory-mapped areas that the kernel allows a process to have. If it is unknown, returns -1.
For more info, check out [virtual memory section of the docs](/fundamentals/storage-memory-usage#virtual-memory). | From a0c727ee7ba085e5d3e739061a6e0a1ed3dfc96c Mon Sep 17 00:00:00 2001 From: Josip Mrden Date: Wed, 29 May 2024 12:51:07 +0200 Subject: [PATCH 2/4] Update storage info where needed --- pages/database-management/server-stats.md | 3 ++- pages/fundamentals/storage-memory-usage.mdx | 16 +++++++++------- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/pages/database-management/server-stats.md b/pages/database-management/server-stats.md index 3f9d42fee..cd11030c4 100644 --- a/pages/database-management/server-stats.md +++ b/pages/database-management/server-stats.md @@ -29,14 +29,15 @@ The result will contain the following fields: | Field | Description | |------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| name | Name of the current database. | | vertex_count | The number of stored nodes (vertices). | | edge_count | The number of stored relationships (edges). | | average_degree | The average number of relationships of a single node. | +| vm_max_map_count | The number of memory-mapped areas that the kernel allows a process to have. If it is unknown, returns -1.
For more info, check out [virtual memory section of the docs](/fundamentals/storage-memory-usage#virtual-memory). | | memory_res | The non-swapped physical RAM memory a task has used, reported by the OS (in B, KiB, MiB, GiB or TiB). | | peak_memory_res | Peak RAM memory usage in the system during the whole run. | | disk_usage | The amount of disk space used by the data directory (in B, KiB, MiB, GiB or TiB). | | memory_tracked | The amount of RAM allocated in the system and tracked by Memgraph (in B, KiB, MiB, GiB or TiB).
For more info, check out [memory control](/fundamentals/storage-memory-usage). | -| vm_max_map_count | The number of memory-mapped areas that the kernel allows a process to have. If it is unknown, returns -1.
For more info, check out [virtual memory section of the docs](/fundamentals/storage-memory-usage#virtual-memory). | | allocation_limit | The current allocation limit set for this instance (in B, KiB, MiB, GiB or TiB).
For more info, check out the [memory control](/fundamentals/storage-memory-usage#control-memory-usage). | | global_isolation_level | The current `global` isolation level.
For more info, check out [isolation levels](/fundamentals/transactions#isolation-levels). | | session_isolation_level | The current `session` isolation level. | diff --git a/pages/fundamentals/storage-memory-usage.mdx b/pages/fundamentals/storage-memory-usage.mdx index c417d3fc7..8b00ea8a4 100644 --- a/pages/fundamentals/storage-memory-usage.mdx +++ b/pages/fundamentals/storage-memory-usage.mdx @@ -729,13 +729,15 @@ SHOW STORAGE INFO; | storage info | value | +--------------------------------+--------------------------------+ | "name" | "memgraph" | -| "vertex_count" | 2677 | -| "edge_count" | 11967 | -| "average_degree" | 8.94061 | -| "memory_usage" | "44.32MiB" | -| "disk_usage" | "133.63KiB" | -| "memory_allocated" | "6.74MiB" | -| "allocation_limit" | "15.28GiB" | +| "vertex_count" | 0 | +| "edge_count" | 0 | +| "average_degree" | 0 | +| "vm_max_map_count" | 453125 | +| "memory_res" | "43.16MiB" | +| "peak_memory_res" | "43.16MiB" | +| "disk_usage" | "104.46KiB" | +| "memory_tracked" | "8.52MiB" | +| "allocation_limit" | "58.55GiB" | | "global_isolation_level" | "SNAPSHOT_ISOLATION" | | "session_isolation_level" | "" | | "next_session_isolation_level" | "" | From 2cabff40d1fe48184d67580bc505bf8e76ccc9dd Mon Sep 17 00:00:00 2001 From: Josip Mrden Date: Wed, 29 May 2024 13:35:48 +0200 Subject: [PATCH 3/4] Update best practics --- pages/data-migration/best-practices.mdx | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/pages/data-migration/best-practices.mdx b/pages/data-migration/best-practices.mdx index 0ae14c6b5..4f517743e 100644 --- a/pages/data-migration/best-practices.mdx +++ b/pages/data-migration/best-practices.mdx @@ -540,6 +540,21 @@ of the `--storage-gc-cycle-sec` flag. That means that Memgraph's garbage collector potentially still didn't deallocate unused objects and free the memory. You can free up memory by running the `FREE MEMORY` query. +## Monitoring memory during import + +Often times, the user needs to batch transactions in the `IN_MEMORY_TRANSACTIONAL` storage mode. +This is due to large-update transactions that are creating a lot of `Delta` objects during the +transaction lifetime. `Delta` objects are necessary for reverting the query if an error during +query execution happens. They are approximately of 56 bytes in size, but large-update transactions +can make the number of delta objects be huge - therefore resulting in Memgraph getting out of memory. + +We have exposed the `peak_memory_res` variable in the `SHOW STORAGE INFO;` command, with which you +can monitor when the peak memory rises in the system. It will help you diagnose bottlenecks, and queries +which utilize a lot of memory, and can be optimized. That is most common in import queries because users +would want to import the whole dataset with one command only. + +For more information, check our [storage memory usage](/fundamentals/storage-memory-usage). + ## Do you have more questions? Schedule a 30 min session with one of our engineers to discuss how Memgraph fits From 0b05e6117953223ab491d377390bda2cab259d47 Mon Sep 17 00:00:00 2001 From: Josip Mrden Date: Mon, 17 Jun 2024 14:56:32 +0200 Subject: [PATCH 4/4] Add information from PR review --- pages/data-migration/best-practices.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pages/data-migration/best-practices.mdx b/pages/data-migration/best-practices.mdx index 4f517743e..735e0f10e 100644 --- a/pages/data-migration/best-practices.mdx +++ b/pages/data-migration/best-practices.mdx @@ -548,10 +548,10 @@ transaction lifetime. `Delta` objects are necessary for reverting the query if a query execution happens. They are approximately of 56 bytes in size, but large-update transactions can make the number of delta objects be huge - therefore resulting in Memgraph getting out of memory. -We have exposed the `peak_memory_res` variable in the `SHOW STORAGE INFO;` command, with which you -can monitor when the peak memory rises in the system. It will help you diagnose bottlenecks, and queries -which utilize a lot of memory, and can be optimized. That is most common in import queries because users -would want to import the whole dataset with one command only. +We have exposed the `peak_memory_res` (peak resident memory) variable in the `SHOW STORAGE INFO;` command. with which you +can monitor when the peak resident memory rises in the system. +It will help you diagnose bottlenecks and high-memory queries which can be optimized. +That is most common in import queries because users would want to import the whole dataset with one command only. For more information, check our [storage memory usage](/fundamentals/storage-memory-usage).