Memory leak in broker module webui #1597
Comments
|
Please post it in the webui repository as it's related to the module. (https://github.com/shinken-monitoring/mod-webui) On Tue, Apr 28, 2015 at 6:51 AM, Ning Xie notifications@github.com wrote:
|
|
But, it seems that the root reason is that the regenerator module has something wrong. Currently, i can tell that when we reload configuration, then the this will cause all the tags count number increase every time we reload the configuration. |
|
tags is a very small list of strings, this won't take up to 30GB unless you I more think about not-cleaned hosts or services. Such objects are far On Tue, Apr 28, 2015 at 9:58 AM, Ning Xie notifications@github.com wrote:
|
|
That's right. I also think so. However, i think WebUI is just an UI, all the data it needs is stored in Can you give me some other threads about how WebUI store the data it needs to do a present? Eventhough the |
sounds like at least fair / probable or possible.. to be investigated |
|
could you confirm or give more context / infos about your setup :
thx for the infos that'll ease the investigation. |
|
I had a problem today that made me think about the broker memory leak issued by @andyxning ! One of the broker processes in my configuration had heay memory consumption : 16 Gb ... In fact, an external module of my broker configuration did throw an exception as soon as ti received a check service brok ... and then it was restarted by the main broker daemon that increased memory consumption ! Thanks to @ddurieux and @Seb-Solon we found how to identify this module and fix the exception ... and now everything seems ok ! @andyxning : check in your brokerd.log if youd do not have such a module that is regularly restarted ? In your top, identify the PID of the process ... take care that the main broker will have heavy memory but there also will be another process that will be launched at every exception ! Try to get the PID of this process and you will be able to grep this PID in the brokerd.log to identify tha bad module |
|
@gst , Our setup info is:
Below is a graph that shows how the memory is used. The down of the line is that we restart the broker daemon and thus restart webui. |
|
We met the same problem this morning ! I do confirm that the master branch WebUI seems to have heavy memory leaks ! Still on survey on my production ... my previous analysis was obviously not that relevant :-( |
|
when using |
|
@andyxning In the mean time you got the workaround : restart the broker say once every 2 or 3 days. But I think we should continue this thread on shinken-monitoring/mod-webui#128 for now. |
|
That's right, i like WebUI for its simplicity and usefulness, though it has Currently, we did restart the broker once every one week, but it is just a ++++++ 2015-05-07 21:44 GMT+08:00 Grégory Starck notifications@github.com:
|
|
As far as i can tell is that the notificationways is duplicated between each shinken reload. |
|
i can now tell that there is something wrong when processing contacts and notificationways in regenerator.py. We only clear hosts and services when we reload the configuration or when one scheduler has down and the spare one takes over the configuration, however, we lost the contacts and notificationways. we initial new notificationways wrongly after the The solution to this problem is very simple, we can just add and flag, indicating whether the notificationway is not, and then initialize(the first time with a In order to further make the contacts and notificationways update with each I think i can make a PR later. |
|
Thanks for the materials about Python memory leaks. ++++++ 2015-05-13 22:26 GMT+08:00 Grégory Starck notifications@github.com:
|
|
Seems that there is still a leak somewhere |
|
Is someone still working on solving this problem. If we can not solve this, Shinken may not scale to large setups. |
|
Hello, Do you have any leaks without reloading arbiter for new configuration Olivier 2015-05-25 15:40 GMT+02:00 Ning Xie notifications@github.com:
|
|
In our Shinken setup, we will change the configuration at sometime. In order to make the new configuration come into effect, we must reload However, with runing Shinken2.4 for at least one week and almost 170 times ++++++ 2015-05-26 15:49 GMT+08:00 Olivier H notifications@github.com:
|
|
There a pull request about this. |
We use WebUI as the frontend of Shinken, however, after about a month later, we find that the memory of WebUI is holding almost 30G memory.
We do not restart Broker daemon meantime. We only do some reload of arbiter if the configuration has changed. So, any threads about how this comes?
The text was updated successfully, but these errors were encountered: