-
Notifications
You must be signed in to change notification settings - Fork 239
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High goroutine count after high client/message load #462
Comments
I've checked all the goroutine calls and it seems Mochi is properly stopping them all. That said, I believe what we've observed must be something related to the two
Therefore, I wonder whether closing the connection is blocked for a long while. I came across this discussion and am now wondering whether Mochi is missing a call to I believe that would explain what happened in our scenario, as we've had tens of thousands of inflight messages according to the HTTP stats endpoint when we disconnected all the clients. |
Mochi can’t immediately detect an ungraceful client disconnect. To do so, it employs two mechanisms:
Both |
We've set the keep-alive to 10 seconds (with great success in general), so I expect the MQTT timeout would kick in after ~15 seconds. However, we've observed multiple minutes of the high goroutine count. Also: The The fact that the connected client count reduced, but the thread count didn't is what gives me pause - is that matching what you just said? |
Given you saw the connected client count reduced shortly (~15s based on the keep-alive of 10s), there should be nothing holding up attachClient from exiting. There are 3 deferred for that function in the actual execution order : defer s.Info.ClientsConnected.Add(-1) // reduce the connected client count, which you saw the effect
defer cl.Stop(nil) // should returned immediately, as in your case it was called once before already
defer s.Listeners.ClientsWg.Done() // would complete immediately One possible cause of what you observed is Threads count do not update in real-time. Can you share the value you used for SysTopicResendInterval? |
We use the default, which is 1 second, I believe. We're hoping to run some more load-tests today and I will report back with the findings. Thanks for your help so far! Much appreciated ❤ |
Hey there! Thanks for this great MQTT broker. I'm a newbie to Go, so please bear with me while I explain the following 😄
@SimonMacIntyre and I have been doing some load-tests and noticed that the
threadsgoroutine count count reported by the HTTP stats endpoint doesn't go down once clients disconnect (ungracefully).This is what we've roughly been doing in this particular test:
Once all clients were connected, we've reached a number of threads. That thread count then stays that high even after the clients were all disconnected (ungracefully).
We've checked in 2 hours later, and the thread count was all the way down to what it was before the test. We instead expected the thread count to go down right after all the clients were shown as disconnected according to Mochi. We can't tell when exactly the thread count went down, but we can confirm it stayed high for at least 10 minutes after all clients were disconnected.
Does anyone have any idea as to what's happening? Is this a cause for concern?
The text was updated successfully, but these errors were encountered: