[1.x] Websocket and admin on HTTP doesn't respond #3062
Labels
Comments
|
As explained in the guidelines, please DON'T paste huge logs inline. I edited your post so that they're collapsed using Github's markdown instead. |
|
I think the two issues are not correlated.
That seems to suggest a missing response from the core/plugin (in this case the request was a
According to libwebsockets internal docs about the reason 38, that is a close from the peer So you need to double-check the code on the other side of the connection here. |
|
Hello,
The problem is, if you check for wss log for both But when Janus is stuck. It gets the request which it is logged but no response. |
Maybe your client is detecting a ping-pong timeout and is closing the connection after 5-6 seconds. However I can tell from the logs that janus received a create session request but never replied, so that means that probably something is locked in the core. Wait until the issue shows up again and provide the output of that should dump the status of all the threads. |
|
Will get back to you with the log, as soon as we get face the issue again. |
|
Hello, here is the output for gdb -p $(pidof janus) --batch -ex "set print pretty on" -ex "set pagination off" -ex "thread apply all bt full" |
|
I can spot several threads blocked, waiting for a mutex:
Indeed it looks like a deadlock. I guess we should enable mutex debug to gather more data. |
|
Hello, |
|
Thanks for the logs. As a side note, the deadlock was triggered by the reception of G.711 packets with unexpected size, as confirmed by your logs: The audiobridge plugin will drop incoming g711 packets when their length is not 160. |
mwalbeck
pushed a commit
to mwalbeck/docker-janus-gateway
that referenced
this issue
Oct 4, 2022
This PR contains the following updates: | Package | Update | Change | |---|---|---| | [meetecho/janus-gateway](https://github.com/meetecho/janus-gateway) | minor | `v1.0.4` -> `v1.1.0` | --- ### Release Notes <details> <summary>meetecho/janus-gateway</summary> ### [`v1.1.0`](https://github.com/meetecho/janus-gateway/blob/HEAD/CHANGELOG.md#v110---2022-10-03) [Compare Source](meetecho/janus-gateway@v1.0.4...v1.1.0) - Added versioning to .so files \[[PR-3075](meetecho/janus-gateway#3075)] - Allow plugins to specify msid in SDPs \[[PR-2998](meetecho/janus-gateway#2998)] - Fixed broken RTCP timestamp on 32bit architectures \[[Issue-3045](meetecho/janus-gateway#3045)] - Fixed problems compiling against recent versions of libwebsockets \[[Issue-3039](meetecho/janus-gateway#3039)] - Updated deprecated DTLS functions to OpenSSL v3.0 [PR-3048](meetecho/janus-gateway#3048)] - Switched to SHA256 for signing self signed DTLS certificates (thanks [@​tgabi333](https://github.com/tgabi333)!) \[[PR-3069](meetecho/janus-gateway#3069)] - Started using strnlen to optimize performance of some strlen calls (thanks [@​tmatth](https://github.com/tmatth)!) \[[PR-3059](meetecho/janus-gateway#3059)] - Added checks to avoid RTX payload type collisions \[[PR-3080](meetecho/janus-gateway#3080)] - Added new APIs for cascading VideoRoom publishers \[[PR-3014](meetecho/janus-gateway#3014)] - Fixed deadlock when using legacy switch in VideoRoom \[[Issue-3066](meetecho/janus-gateway#3066)] - Fixed disabled property not being advertized to subscribers when VideoRoom publishers removed tracks - Fixed occasional deadlock when using G.711 in the AudioBridge \[[Issue-3062](meetecho/janus-gateway#3062)] - Added new way of capturing devices/tracks in janus.js \[[PR-3003](meetecho/janus-gateway#3003)] - Removed call to .stop() for remote tracks in demos \[[PR-3056](meetecho/janus-gateway#3056)] - Fixed missing message/info/transfer buttons in SIP demo page - Fixed postprocessing compilation issue on older FFmpeg versions \[[PR-3064](meetecho/janus-gateway#3064)] - Other smaller fixes and improvements (thanks to all who contributed pull requests and reported issues!) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzMi4yMTMuMCIsInVwZGF0ZWRJblZlciI6IjMyLjIxMy4wIn0=--> Reviewed-on: https://git.walbeck.it/walbeck-it/docker-janus-gateway/pulls/91 Co-authored-by: renovate-bot <bot@walbeck.it> Co-committed-by: renovate-bot <bot@walbeck.it>
|
We'll assume this is fixed, due to lack of feedback. Closing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What version of Janus is this happening on?
v1.0.3
"commit-hash": "95bfcf35969894dd3361b6f2d65fd6a453139b11"OS: Redhat 7.5
Have you tested a more recent version of Janus too?
No, This was latest version when we deployed to production
Was this working before?
Didn't face it before.
Description
Recently very frequently we are facing this issue, Websocket and admin HTTP API stops responding but the janus-gw keeps running.
It is happening randomly.
I will provide 2 logs, one with the issue and one without the issue.
Log With Issue
Log Without Issue
The text was updated successfully, but these errors were encountered: