crash when querying admin's handle_info

[kalturaguy]

Hi,

In the multi-stream branch, there is crash if we call handle_info during establishing connection to room plugin.

Reproduced 100% each time

Thanks,

Dump is in here:
https://januscrashdumps.s3.eu-central-1.amazonaws.com/dump.tar.gz

Call stack

Thread 24 (Thread 0x7f858affd700 (LWP 29)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f85d941e25a in g_cond_wait_until () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93a60c1 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f85d93a6681 in g_async_queue_timeout_pop () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f85d93fcf11 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 23 (Thread 0x7f858bfff700 (LWP 27)):
#0  0x00007f85d8a80819 in __GI___poll (fds=0x7f85a4ffa010, nfds=4, timeout=30000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f85d41fe9f2 in _lws_plat_service_tsi () from /usr/lib/libwebsockets.so.17
#2  0x00007f85d41febd7 in lws_plat_service () from /usr/lib/libwebsockets.so.17
#3  0x00007f85d42189f7 in lws_service () from /usr/lib/libwebsockets.so.17
#4  0x00007f85d424c12f in janus_websockets_thread (data=0x55a472b2c390) at transports/janus_websockets.c:1330
#5  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 22 (Thread 0x7f85bf7fe700 (LWP 16)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f85d941e13f in g_cond_wait () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93a60db in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f85d93a6507 in g_async_queue_pop () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f85d462137a in janus_videoroom_handler (data=<optimized out>) at plugins/janus_videoroom.c:7056
--Type <RET> for more, q to quit, c to continue without paging--c
#5  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 21 (Thread 0x7f858b7fe700 (LWP 28)):
#0  0x00007f85d8a80819 in __GI___poll (fds=0x7f85c4001720, nfds=2, timeout=82) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f85d93d41f6 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93d4582 in g_main_loop_run () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x000055a471bb796b in janus_ice_handle_thread (data=0x7f85c4001ae0) at ice.c:1219
#4  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 20 (Thread 0x7f85be7fc700 (LWP 18)):
#0  0x00007f85d8b64bf0 in __GI___nanosleep (requested_time=0x7f85be7fb130, remaining=0x7f85be7fb140) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x00007f85d93fdb7f in g_usleep () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d45b7b2a in janus_streaming_filesource_thread (data=0x55a472a83510) at plugins/janus_streaming.c:8149
#3  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7f85beffd700 (LWP 17)):
#0  0x00007f85d8b64bf0 in __GI___nanosleep (requested_time=0x7f85beffc130, remaining=0x7f85beffc140) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x00007f85d93fdb7f in g_usleep () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d45b7b2a in janus_streaming_filesource_thread (data=0x55a472a82180) at plugins/janus_streaming.c:8149
#3  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7f85d7e761c0 (LWP 7)):
#0  0x00007f85d8a80819 in __GI___poll (fds=0x55a472a71be0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f85d93d41f6 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93d4582 in g_main_loop_run () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x000055a471b92d3e in main (argc=<optimized out>, argv=<optimized out>) at janus.c:5470

Thread 17 (Thread 0x7f85a7fff700 (LWP 22)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f85d941e13f in g_cond_wait () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93a60db in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f85d93a6507 in g_async_queue_pop () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f85d43875da in janus_audiobridge_handler (data=<optimized out>) at plugins/janus_audiobridge.c:5437
#5  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7f85bcff9700 (LWP 21)):
#0  0x00007f85d8b64bf0 in __GI___nanosleep (requested_time=0x7f85bcff5360, remaining=0x7f85bcff5370) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x00007f85d93fdb7f in g_usleep () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d4382bca in janus_audiobridge_mixer_thread (data=<optimized out>) at plugins/janus_audiobridge.c:7146
#3  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7f858a7fc700 (LWP 30)):
#0  0x00007f85d8a80819 in __GI___poll (fds=0x7f85c4004ea0, nfds=2, timeout=83) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f85d93d41f6 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93d4582 in g_main_loop_run () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x000055a471bb796b in janus_ice_handle_thread (data=0x7f85c4005100) at ice.c:1219
#4  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7f85bdffb700 (LWP 19)):
#0  0x00007f85d8a80819 in __GI___poll (fds=fds@entry=0x7f85bdffa310, nfds=nfds@entry=3, timeout=timeout@entry=1000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f85d45b9479 in janus_streaming_relay_thread (data=0x55a472a81e70) at plugins/janus_streaming.c:8417
#2  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#4  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f85a5ffb700 (LWP 26)):
#0  0x00007f85d8a8b7ef in epoll_wait (epfd=19, events=0x7f85a5ff9f60, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007f85d42697ca in ?? () from /usr/lib/x86_64-linux-gnu/libmicrohttpd.so.12
#2  0x00007f85d426aab7 in ?? () from /usr/lib/x86_64-linux-gnu/libmicrohttpd.so.12
#3  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#4  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f85d6649700 (LWP 11)):
#0  __libc_recvmsg (flags=0, msg=0x7f85d6648300, fd=5) at ../sysdeps/unix/sysv/linux/recvmsg.c:28
#1  __libc_recvmsg (fd=5, msg=0x7f85d6648300, flags=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:25
#2  0x00007f85d8e8e8ff in recv_function_raw6 (arg=0x0) at user_recv_thread.c:548
#3  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#4  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f85a67fc700 (LWP 25)):
#0  0x00007f85d8a80819 in __GI___poll (fds=0x55a472ab2770, nfds=1, timeout=10000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f85d93d41f6 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93d4582 in g_main_loop_run () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f85d4280d9d in janus_http_timer (user_data=<optimized out>) at transports/janus_http.c:380
#4  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f85d5e48700 (LWP 12)):
#0  0x00007f85d8b64bf0 in __GI___nanosleep (requested_time=0x7f85d5e47ba0, remaining=0x7f85d5e47b90) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x00007f85d8ea35ef in user_sctp_timer_iterate (arg=0x0) at netinet/sctp_callout.c:209
#2  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#3  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f85a77fe700 (LWP 23)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f85d941e13f in g_cond_wait () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93a60db in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f85d93a6507 in g_async_queue_pop () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f85d42e4609 in janus_lua_scheduler (data=<optimized out>) at plugins/janus_lua.c:2533
#5  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f85d6e4a700 (LWP 10)):
#0  __libc_recvmsg (flags=0, msg=0x7f85d6e49af0, fd=4) at ../sysdeps/unix/sysv/linux/recvmsg.c:28
#1  __libc_recvmsg (fd=4, msg=0x7f85d6e49af0, flags=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:25
#2  0x00007f85d8e8e307 in recv_function_raw (arg=0x0) at user_recv_thread.c:349
#3  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#4  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f85d7e6d700 (LWP 8)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f85d941e13f in g_cond_wait () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x000055a471bd7a37 in janus_log_thread (ctx=<optimized out>) at log.c:106
#3  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f85d764b700 (LWP 9)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f85d8f40a98 <sctp_it_ctl+120>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f85d8f40a20 <sctp_it_ctl>, cond=0x7f85d8f40a70 <sctp_it_ctl+80>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f85d8f40a70 <sctp_it_ctl+80>, mutex=0x7f85d8f40a20 <sctp_it_ctl>) at pthread_cond_wait.c:655
#3  0x00007f85d8ea2dad in sctp_iterator_thread (v=0x0) at netinet/sctp_bsd_addr.c:149
#4  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f85d5647700 (LWP 13)):
#0  0x00007f85d8a80819 in __GI___poll (fds=0x55a472a72880, nfds=1, timeout=1998) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f85d93d41f6 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93d4582 in g_main_loop_run () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x000055a471bc4fc0 in janus_sessions_watchdog (user_data=0x55a472a714e0) at janus.c:726
#4  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f85a6ffd700 (LWP 24)):
#0  0x00007f85d8a80819 in __GI___poll (fds=0x55a472aad680, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f85d93d41f6 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93d4582 in g_main_loop_run () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f85d42e43a2 in janus_lua_timer (data=0x55a472aae740) at plugins/janus_lua.c:2558
#4  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f85bd7fa700 (LWP 20)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f85d941e13f in g_cond_wait () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93a60db in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f85d93a6507 in g_async_queue_pop () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f85d45bcfaa in janus_streaming_handler (data=<optimized out>) at plugins/janus_streaming.c:5237
#5  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f85bffff700 (LWP 15)):
#0  0x00007f85d8a80819 in __GI___poll (fds=0x55a472a85190, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f85d93d41f6 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f85d93d4582 in g_main_loop_run () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f85d45faee1 in janus_videoroom_rtp_forwarder_rtcp_thread (data=<optimized out>) at plugins/janus_videoroom.c:10269
#4  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f85d4e46700 (LWP 14)):
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:96
#1  0x00007f85d937f8be in json_string () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#2  0x00007f85d45fe905 in janus_videoroom_query_session (handle=<optimized out>) at plugins/janus_videoroom.c:3746
#3  janus_videoroom_query_session (handle=<optimized out>) at plugins/janus_videoroom.c:3691
#4  0x000055a471bd0fa6 in janus_process_incoming_admin_request (request=0x7f858400b890) at janus.c:2839
#5  0x000055a471bd3c57 in janus_transport_requests (data=<optimized out>) at janus.c:3356
#6  0x00007f85d93fc4d5 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#7  0x00007f85d8b5afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007f85d8a8b4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[lminiero]

As clearly explained in the guidelines, please do NOT paste huge chunks of text inline: yours even messed with issues references. I edited the text for you to make it collapsible, this one time.

[lminiero]

You should also write exactly which commit you're on, or line numbers may not make sense.

[lminiero]

The above commit fixes it for me.