Windows: free disk space monitor logs timeouts #13759
-
Community Support Policy
RabbitMQ version usedother (please specify) Erlang version used26.2.x Operating system (distribution) usedWindows How is RabbitMQ deployed?Windows installer rabbitmq-diagnostics status outputSee https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics Logs from node 1 (with sensitive values edited out)RabbitMQ log is given below => 2025-03-21 09:31:04.682000+00:00 [info] <0.409.0> Free disk space monitor was already disabled Erlang Crash dump information is given below => =erl_crash_dump:0.5 Logs from node 2 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 3 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs rabbitmq.confNA Steps to deploy RabbitMQ clusterNA Steps to reproduce the behavior in questionNA advanced.configSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location Application code# PASTE CODE HERE, BETWEEN BACKTICKSKubernetes deployment file# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKSWhat problem are you trying to solve?Erlang is generating timeout exception, triggers shutdown and create a erlang crashdump. We are using RabbitMQ v3.13.2 along with Erlang/OTP 26.2.3 for windows platform. Why are we getting timeout exception and what are the solution. |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 4 replies
-
|
This timeout typically means that the OS monitor’s disk checks are timing out, even with deactivated free disk monitoring. Windows can sometimes delay disk info responses, causing os_mon’s get_disk_info call to exceed its timeout. A possible solution is to further disable or adjust os_mon parameters in your advanced.config (or disable os_mon entirely for disk metrics) or upgrade to a version of RabbitMQ/Erlang with improved Windows support. This issue has been noted on Windows installations, so reviewing any known issues in RabbitMQ’s GitHub or support channels may also help. |
Beta Was this translation helpful? Give feedback.
-
|
How to disable os_mon entirely. |
Beta Was this translation helpful? Give feedback.
-
|
@sharad578 are you sure it's a good idea to disable it? Have you overprovisioned your disks as recommended? Sorry but I'd bet you did not. There are rabbitmqctl.bat deactivate_free_disk_space_monitoringrabbitmqctl.bat activate_free_disk_space_monitoringwhich disable and re-enable the free disk space monitor on the target node. Note that after a node restart they have to be run again, there is no way to disable this monitoring permanently. Running RabbitMQ on something other than Windows should also help, e.g. in an OCI container. |
Beta Was this translation helpful? Give feedback.
-
|
@sharad578 your node reports that the partition where its data directory is stored is almost full ( |
Beta Was this translation helpful? Give feedback.
-
|
@sharad578 I do not see how these timeouts could result in a node termination. It's a known behavior in some Windows environments, and the node continues running in every other report. I guess we'll have to introduce a way to opt-in to disable this feature entirely but we know for a fact that most users do not have reasonable monitoring in place, and monitoring free disk space is critically important. |
Beta Was this translation helpful? Give feedback.
-
|
disk_almost_full alarm is set for P:. Note that this drive is not used RabbitMQ in our environment. RabbitMQ uses E:\ which has enough space. It's not clear why get_disk_info api is called even if disk space monitoring is disabled using command rabbitmqctl.bat deactivate_free_disk_space_monitoring relevant logs snippet => 2025-03-21 09:31:04.682000+00:00 [info] <0.409.0> Free disk space monitor was already disabled |
Beta Was this translation helpful? Give feedback.
-
|
That's the runtime free disk space monitoring logging that, not RabbitMQ. Whether there is an easy reliable way to disable it for a specific drive, remains to be seen. According to the docs, all "fixed" drives are monitored, and you can only disable disk space monitoring entirely, in RabbitMQ's case via [
{os_mon, [
{start_disksup, false}
]}
].(the trailing dot is crucially important, as with any Erlang term file) We know for a fact that detecting what partition the node has its data directory on is not as trivial as it sounds. With the idea of moving different parts of node data to different drives (e.g. streams, QQs) being more and more relevant, this will only get worse. |
Beta Was this translation helpful? Give feedback.
@sharad578 are you sure it's a good idea to disable it? Have you overprovisioned your disks as recommended? Sorry but I'd bet you did not.
There are
rabbitmqctl.bat deactivate_free_disk_space_monitoringrabbitmqctl.bat activate_free_disk_space_monitoringwhich disable and re-enable the free disk space monitor on the target node. Note that after a node restart they have to be run again, there is no way to disable this monitoring permanently.
Running RabbitMQ on something other than Windows should also help, e.g. in an OCI container.