Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WebRTC: SRS 5 docker crash when RTMP2RTC over TCP. #3784

Closed
namnm opened this issue Aug 26, 2023 · 5 comments
Closed

WebRTC: SRS 5 docker crash when RTMP2RTC over TCP. #3784

namnm opened this issue Aug 26, 2023 · 5 comments
Assignees
@winlinvip
Labels
EnglishNative This issue is conveyed exclusively in English. WebRTC WebRTC, RTC2RTMP or RTMP2RTC.

Comments

@namnm
Copy link

namnm commented Aug 26, 2023

Note: Please read FAQ before file an issue, see #2716

Description

While I was testing with rtmp to webrtc, the webrtc stream worked well. Then I refreshed the browser, and srs crashed

  1. SRS Version: 5

  2. SRS Log:

[2023-08-26 02:12:23.990][INFO][1][85uf3e42] HTTP #0 172.19.0.1:43648 GET http://localhost:8080/players/rtc_player.html?autostart=true, content-length=-1
[2023-08-26 02:12:23.991][INFO][1][85uf3e42] http match file=./objs/nginx/html/players/rtc_player.html, pattern=/, upath=/players/rtc_player.html
[2023-08-26 02:12:23.993][INFO][1][85uf3e42] TCP: before dispose resource(HttpConn)(0x60700005e7b0), conns=3, zombies=0, ign=0, inz=0, ind=0
[2023-08-26 02:12:23.993][WARN][1][85uf3e42][104] client disconnect peer. ret=1007
[2023-08-26 02:12:23.993][INFO][1][95fw2y11] TCP: clear zombies=1 resources, conns=3, removing=0, unsubs=0
[2023-08-26 02:12:23.993][INFO][1][85uf3e42] TCP: disposing #0 resource(HttpConn)(0x60700005e7b0), conns=3, disposing=1, zombies=0
[2023-08-26 02:12:24.004][INFO][1][5114w0fh] DTLS: After done, got 39 bytes
[2023-08-26 02:12:24.004][INFO][1][5114w0fh] DTLS: State Passive RECV, done=1, arq=0, r0=39, len=39, cnt=21, size=26, hs=0
[2023-08-26 02:12:24.005][WARN][1][5114w0fh][0] DTLS: SSL3 alert method=read type=warning, desc=CN(close notify), where=16388, ret=256, r1=0
[2023-08-26 02:12:24.005][INFO][1][5114w0fh] RTC: session destroy by DTLS alert(warning CN), username=p81x7808:tctP
[2023-08-26 02:12:24.005][INFO][1][5114w0fh] RTC: before dispose resource(RtcConn)(0x61d000194680), conns=1, zombies=0, ign=0, inz=0, ind=0
[2023-08-26 02:12:24.005][INFO][1][5114w0fh] RTC: session detach from [5114w0fh](RtcConn), disposing=1
[2023-08-26 02:12:24.005][INFO][1][5114w0fh] RTC: tcp conn diposing, because of rtc connection
[2023-08-26 02:12:24.005][INFO][1][396m2q82] TCP: before dispose resource(Tcp)(0x60c0000a4b00), conns=2, zombies=0, ign=0, inz=0, ind=0
[2023-08-26 02:12:24.005][ERROR][1][5114w0fh][0] serve error code=1070(StThreadInterrupt)(ST thread is interrupted) : rtc tcp conn : interrupted
thread [1][5114w0fh]: do_cycle() [./src/app/srs_app_rtc_network.cpp:811][errno=0]
thread [1][5114w0fh]: interrupt() [./src/app/srs_app_st.cpp:257][errno=0]
[2023-08-26 02:12:24.005][INFO][1][e0i043rg] RTC: clear zombies=1 resources, conns=1, removing=0, unsubs=2
[2023-08-26 02:12:24.005][INFO][1][5114w0fh] RTC: disposing #0 resource(RtcConn)(0x61d000194680), conns=1, disposing=1, zombies=0
[2023-08-26 02:12:24.005][INFO][1][95fw2y11] TCP: clear zombies=1 resources, conns=2, removing=0, unsubs=0
[2023-08-26 02:12:24.005][INFO][1][396m2q82] TCP: disposing #0 resource(Tcp)(0x60c0000a4b00), conns=2, disposing=1, zombies=0
=================================================================
==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x6040013a9358 at pc 0x5646a13fec4f bp 0x7f57effffd60 sp 0x7f57effffd50
READ of size 8 at 0x6040013a9358 thread T1 (srs-hybrid-2)
    #0 0x5646a13fec4e in SrsRtcTcpNetwork::write(void*, unsigned long, long*) src/app/srs_app_rtc_network.cpp:672
    #1 0x5646a1389e3b in SrsRtcConnection::do_send_packet(SrsRtpPacket*) src/app/srs_app_rtc_conn.cpp:2472
    #2 0x5646a143e0c2 in SrsRtcAudioSendTrack::on_rtp(SrsRtpPacket*) src/app/srs_app_rtc_source.cpp:2804
    #3 0x5646a1374461 in SrsRtcPlayStream::send_packet(SrsRtpPacket*&) src/app/srs_app_rtc_conn.cpp:734
    #4 0x5646a1373254 in SrsRtcPlayStream::cycle() src/app/srs_app_rtc_conn.cpp:669
    #5 0x5646a10f5e93 in SrsFastCoroutine::cycle() src/app/srs_app_st.cpp:285
    #6 0x5646a10f5fe3 in SrsFastCoroutine::pfn(void*) src/app/srs_app_st.cpp:300
    #7 0x5646a14a70c9 in _st_thread_main /srs/trunk/objs/Platform-SRS5-Linux-5.15.0-GCC9.4.0-x86_64/st-srs/sched.c:380
    #8 0x5646a14a79ef in st_thread_create /srs/trunk/objs/Platform-SRS5-Linux-5.15.0-GCC9.4.0-x86_64/st-srs/sched.c:666

0x6040013a9358 is located 8 bytes inside of 48-byte region [0x6040013a9350,0x6040013a9380)
freed by thread T1 (srs-hybrid-2) here:
    #0 0x5646a0ddbf6f in operator delete(void*) (/usr/local/srs/objs/srs+0x4aef6f)

previously allocated by thread T1 (srs-hybrid-2) here:
    #0 0x5646a0ddafd7 in operator new(unsigned long) (/usr/local/srs/objs/srs+0x4adfd7)

Thread T1 (srs-hybrid-2) created by T0 here:
    #0 0x5646a0d06295 in pthread_create (/usr/local/srs/objs/srs+0x3d9295)
    #1 0x5646a1333697 in SrsThreadPool::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SrsCplxError* (*)(void*), void*) src/app/srs_app_threads.cpp:671
    #2 0x5646a14a59fe in run_in_thread_pool() src/main/srs_main_server.cpp:517
    #3 0x5646a14a5466 in run_directly_or_daemon() src/main/srs_main_server.cpp:456
    #4 0x5646a14a25c2 in do_main(int, char**, char**) src/main/srs_main_server.cpp:245
    #5 0x5646a14a28dd in main src/main/srs_main_server.cpp:256
    #6 0x7f57f7aa5082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082)

SUMMARY: AddressSanitizer: heap-use-after-free src/app/srs_app_rtc_network.cpp:672 in SrsRtcTcpNetwork::write(void*, unsigned long, long*)
Shadow bytes around the buggy address:
  0x0c088026d210: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
  0x0c088026d220: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
  0x0c088026d230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c088026d240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c088026d250: fa fa fa fa fa fa fa fa fa fa fd fd fd fa fa fa
=>0x0c088026d260: fa fa fd fd fd fa fa fa fa fa fd[fd]fd fd fd fd
  0x0c088026d270: fa fa fd fd fd fa fa fa fa fa fa fa fa fa fa fa
  0x0c088026d280: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
  0x0c088026d290: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
  0x0c088026d2a0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
  0x0c088026d2b0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0] =================================================================
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0] ==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x6040013a9358 at pc 0x5646a13fec4f bp 0x7f57effffd60 sp 0x7f57effffd50
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0] READ of size 8 at 0x6040013a9358 thread T1 (srs-hybrid-2)
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #0 0x5646a13fec4e in SrsRtcTcpNetwork::write(void*, unsigned long, long*) src/app/srs_app_rtc_network.cpp:672, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #1 0x5646a1389e3b in SrsRtcConnection::do_send_packet(SrsRtpPacket*) src/app/srs_app_rtc_conn.cpp:2472, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #2 0x5646a143e0c2 in SrsRtcAudioSendTrack::on_rtp(SrsRtpPacket*) src/app/srs_app_rtc_source.cpp:2804, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #3 0x5646a1374461 in SrsRtcPlayStream::send_packet(SrsRtpPacket*&) src/app/srs_app_rtc_conn.cpp:734, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #4 0x5646a1373254 in SrsRtcPlayStream::cycle() src/app/srs_app_rtc_conn.cpp:669, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #5 0x5646a10f5e93 in SrsFastCoroutine::cycle() src/app/srs_app_st.cpp:285, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #6 0x5646a10f5fe3 in SrsFastCoroutine::pfn(void*) src/app/srs_app_st.cpp:300, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #7 0x5646a14a70c9 in _st_thread_main /srs/trunk/objs/Platform-SRS5-Linux-5.15.0-GCC9.4.0-x86_64/st-srs/sched.c:380, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0]     #8 0x5646a14a79ef in st_thread_create /srs/trunk/objs/Platform-SRS5-Linux-5.15.0-GCC9.4.0-x86_64/st-srs/sched.c:666, r0=1093
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0] 0x6040013a9358 is located 8 bytes inside of 48-byte region [0x6040013a9350,0x6040013a9380)
[2023-08-26 02:12:24.054][ERROR][1][5114w0fh][0] freed by thread T1 (srs-hybrid-2) here:
sh: 1: addr2line: not found
[2023-08-26 02:12:24.057][ERROR][1][5114w0fh][0]     #0 0x5646a0ddbf6f in operator delete(void*) (/usr/local/srs/objs/srs+0x4aef6f), r0=1094
[2023-08-26 02:12:24.057][ERROR][1][5114w0fh][0] previously allocated by thread T1 (srs-hybrid-2) here:
[2023-08-26 02:12:24.057][ERROR][1][5114w0fh][0]     #0 0x5646a0ddafd7 in operator new(unsigned long) (/usr/local/srs/objs/srs+0x4adfd7), r0=1094
[2023-08-26 02:12:24.057][ERROR][1][5114w0fh][0] Thread T1 (srs-hybrid-2) created by T0 here:
sh: 1: addr2line: not found
sh: 1: addr2line: not found
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #0 0x5646a0d06295 in pthread_create (/usr/local/srs/objs/srs+0x3d9295), r0=1094
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #1 0x5646a1333697 in SrsThreadPool::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SrsCplxError* (*)(void*), void*) src/app/srs_app_threads.cpp:671, r0=1093
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #2 0x5646a14a59fe in run_in_thread_pool() src/main/srs_main_server.cpp:517, r0=1093
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #3 0x5646a14a5466 in run_directly_or_daemon() src/main/srs_main_server.cpp:456, r0=1093
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #4 0x5646a14a25c2 in do_main(int, char**, char**) src/main/srs_main_server.cpp:245, r0=1093
[2023-08-26 02:12:24.058][ERROR][1][5114w0fh][0]     #5 0x5646a14a28dd in main src/main/srs_main_server.cpp:256, r0=1093
sh: 1: addr2line: not found
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]     #6 0x7f57f7aa5082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082), r0=1094
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0] SUMMARY: AddressSanitizer: heap-use-after-free src/app/srs_app_rtc_network.cpp:672 in SrsRtcTcpNetwork::write(void*, unsigned long, long*)
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0] Shadow bytes around the buggy address:
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d210: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d220: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d250: fa fa fa fa fa fa fa fa fa fa fd fd fd fa fa fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0] =>0x0c088026d260: fa fa fd fd fd fa fa fa fa fa fd[fd]fd fd fd fd
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d270: fa fa fd fd fd fa fa fa fa fa fa fa fa fa fa fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d280: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d290: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d2a0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   0x0c088026d2b0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0] Shadow byte legend (one shadow byte represents 8 application bytes):
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Addressable:           00
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Partially addressable: 01 02 03 04 05 06 07 
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Heap left redzone:       fa
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Freed heap region:       fd
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Stack left redzone:      f1
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Stack mid redzone:       f2
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Stack right redzone:     f3
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Stack after return:      f5
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Stack use after scope:   f8
[2023-08-26 02:12:24.061][ERROR][1][5114w0fh][0]   Global redzone:          f9
==1==ABORTING
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Global init order:       f6
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Poisoned by user:        f7
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Container overflow:      fc
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Array cookie:            ac
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Intra object redzone:    bb
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   ASan internal:           fe
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Left alloca redzone:     ca
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Right alloca redzone:    cb
[2023-08-26 02:12:24.062][ERROR][1][5114w0fh][0]   Shadow gap:              cc
  1. SRS Config:
listen 1935;
daemon off;
srs_log_tank console;
# srs_log_level warn;

srt_server {
  enabled on;
  listen 10080;
}

rtc_server {
  enabled on;
  protocol tcp;
  candidate 192.168.5.234;
  tcp {
    enabled on;
    listen 8000;
  }
}
http_api {
  enabled on;
  listen 1985;
  crossdomain on;
}
http_server {
  enabled on;
  listen 8080;
  dir ./objs/nginx/html;
}

vhost __defaultVhost__ {
  rtc {
    enabled on;
    rtmp_to_rtc on;
  }
  http_remux {
    enabled on;
    mount [vhost]/[app]/[stream].flv;
    hstrs on;
  }
}

Replay

While I was testing with rtmp to webrtc, the webrtc stream worked well. Then I refreshed the browser, and srs crashed

Expect

Do not crash
@winlinvip winlinvip added the EnglishNative This issue is conveyed exclusively in English. label Aug 26, 2023
@winlinvip
Copy link
Member

winlinvip commented Aug 28, 2023 •
edited
Loading

What browser are you using, what is your video source, does it always occur? If it does, could you please attach the video source file.

TRANS_BY_GPT4

@winlinvip winlinvip changed the title SRS 5 docker crash WebRTC: SRS 5 docker crash when RTMP2RTC over TCP. Aug 28, 2023
@namnm
Copy link
Author

namnm commented Sep 4, 2023 •
edited
Loading

I used latest GoogleChrome on macOS. It only occurs after refreshing the browser, and not always occur just rarely. About the video source, it happened on several videos I tried, so I guess the issue could be at the browser refreshing which caused an error in the connection. I can see some related info in the above log.

@winlinvip
Copy link
Member

winlinvip commented Sep 5, 2023 •
edited
Loading

When the browser refreshes, it is possible to disconnect the TCP connection and then re-establish a new TCP connection. At this time, it is indeed possible to trigger boundary conditions.

Please spend more time researching under what conditions it is more likely to occur, such as whether it is specific video content, a specific browser, or a specific operation method.

TRANS_BY_GPT4

@DancingPanda666 DancingPanda666 mentioned this issue Jan 8, 2024
Closed
@winlinvip winlinvip self-assigned this Mar 30, 2024
@winlinvip winlinvip added the WebRTC WebRTC, RTC2RTMP or RTMP2RTC. label Mar 30, 2024
@winlinvip winlinvip mentioned this issue Mar 30, 2024
Closed
@winlinvip
Copy link
Member

For #3505 SRS crashed at closing a active fd:

void srs_close_stfd(srs_netfd_t& stfd)
{
    if (stfd) {
        // we must ensure the close is ok.
        int r0 = st_netfd_close((st_netfd_t)stfd);
        if (r0) {
            // By _st_epoll_fd_close or _st_kq_fd_close
            if (errno == EBUSY) srs_assert(!r0);

@winlinvip winlinvip mentioned this issue Mar 30, 2024
Closed
This was referenced Apr 22, 2024
Closed
Open
@winlinvip
Copy link
Member

winlinvip commented Jun 12, 2024 •
edited
Loading

Fixed, see #4083

@winlinvip winlinvip mentioned this issue Jun 12, 2024
Merged
winlinvip added a commit that referenced this issue Jun 13, 2024
@winlinvip @suzp1984
SmartPtr: Use shared ptr in RTC TCP connection. v6.0.127 (#4083)
242152b 
Fix issue #3784

---

Co-authored-by: Jacob Su <suzp1984@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@winlinvip winlinvip

Labels
EnglishNative This issue is conveyed exclusively in English. WebRTC WebRTC, RTC2RTMP or RTMP2RTC.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@winlinvip @namnm