[OH3] 3.0.0M3 has substantial lag in processing and increased CPU #1852
Comments
|
I am on M3 and the java process is at 3% CPU - I don't think there is any general issue in the core. |
|
Happy to. I've never done that before, can you provide a link so I can get started? |
|
Here's all you need to now :-) |
|
Looks fun. I'll drop M3 back on probably Sunday and see what I can capture. I am intrigued that M2 with M3 addons works just fine where M3 has this issue. I looked through the changes for M3 and I didn't see much in terms of things that could impact this. |
|
Not exactly sure what I'm looking at, but starting to dig through it now. Uploading the thread dumps for someone who knows this better than me to look at if they have some time. I updated to M3 via dpkg and immediately noticed the load on the process staying over 100%. I gave the system 10 minutes to "calm down" (it didn't). I ran 3 separate thread dumps during that 10 mins which I'm going through now. I stopped the system and put M2 back in via dpkg. System is running just fine. There were no config changes made at all. Just start/stop of process and dpkg. No cache cleaning, nada. M3-Slow10Min.txt EDIT1: versus M2: Why would the cpu time be on the order of 5+ minutes when only 10 minutes have passed for M3 and barely 1 second after 10 minutes for M2? EDIT2: |
|
@kaikreuzer M4 seems to have the same issue for me. M2 with M4 addons works just fine. Also interesting that it seems to identify AbstractLink as what it's waiting on. I attempted to do the dump at exactly the 10 minute mark as a comparison point. It looks like OH-eventexecutor-1 ran for 7+ minutes out of the 10 which seems excessive. @cweitkamp As you did #1794 I thought you may be interested in this. |
|
Thanks for your analysis, @morph166955, you seem to be becoming a Java thread dump expert very quickly ;-). Indeed #1794 was the only change in that code and it is interesting that the newly added |
|
No problem. It's a fun blast from the past for me. I used to do this on C 15-20 years ago when I was getting my degree. Just haven't done enough Java where I've had to do it on this. I went more networking in the past few years so this has been more of a hobby lately. From a process standpoint, the only thing that I did was "dpkg -i" between M2 and M4 and start/stop of systemctl. EDIT: I could be totally off on a wrong idea here, but I don't know if it's relevant to note that all of my items are configured via text config (nothing done in the jsondb) and the only things I have configured in the GUI are the Chaimberlin garage doors I have (but not the items). Could an empty database be causing this? |
|
I do not see any performance issue with the new While traversing the stack trace provided by @morph166955 - thanks for the research - I figured a line of code in the |
|
I also have a problem with high cpu usage in M2 and M3. It seemed that the rule is parsed every time, because every 5 seconds (5 seconds update period of item E3DC_Extern) I got the log error. Maybe preparsing the rule after saving it in main ui makes the trick. |
@gdolfen That's very likely not related - please see the issue description, which says that this effect is only on M3, but NOT on M2. |
@cweitkamp So are you saying that the same event is coming in twice per linked item (I see every value being listed 8 times and you say that there are 4 items linked)? And it looks weird that it takes 1-2 seconds to process every new value. |
@kaikreuzer |
|
@gdolfen Sounds new to me, so yes, a new bug report for it would be perfect! 👍 |
|
@kaikreuzer Here is the new bug report: [#1877] |
|
I am afraid my observed behavior suffered from a bad binding implementation which sends updates to all channels if a REFRESH was sent: openhab/openhab-addons#9188. //EDIT: I was able to reduce it to two events per Item (see PR). |
|
@kaikreuzer To play Devils advocate, is there a really good reason to not backout the change until we understand why this is causing the issue? That way we can confirm that it is in fact the root cause of the trigger and get the releases moving forward? |
|
I honestly doubt that #1794 is the culprit here - it is probably just a coincidence that the thread dump showed up the |
|
Thats easy enough for me to test. Please let me know what snapshot number has the fix in it and I'll go drop it on to see. |
|
@morph166955 Latest snapshot 2041 has the fix included. |
|
@kaikreuzer I went to https://openhab.jfrog.io/openhab/openhab-linuxpkg/pool/main/3.0.0~S2041/ to pull but it looks like only the addons compiled, there is no main deb. 2039 looks like the last one that compiled the core. |
|
That's correct - there was an Artifactory server update today and @BClark09 is working on getting the linux packages published again... For the time being, only the zip distribution is available indeed! |
|
Gotcha. I'll hold for the deb package. As soon as it's done I'll get it loaded and tested to make sure this is squared away. |
|
All sorted! S2041 is available on the apt repo now. |
|
@BClark09 You're the best! Thanks for the ultra-quick help! 🎉 |
|
@kaikreuzer I think we're good! CPU bouncing between 4% and 50%. Everything looks like it's updating like normal. I'll stay on S2041 to make sure it's stable. THANKS!!! |
|
That's great news, I am VERY relieved to hear that, thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Upgraded from M2 to M3. On M2, the java process sits around 20-30% CPU usage as per "top". M3 sits around 115-145%. I backed out from M3 back to M2 but left the addons as M3 (confirmed with bundle:list -s showing M2 and M3 packages). No issues with M3 addons on M2 from a usage perspective. Running on Ubuntu 20.04 in a (beefy) VM.
The text was updated successfully, but these errors were encountered: