Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

input_thread: mixing input threads and coroutines leads to a crash #7061

Closed
edsiper opened this issue Mar 23, 2023 · 5 comments
Closed

input_thread: mixing input threads and coroutines leads to a crash #7061

edsiper opened this issue Mar 23, 2023 · 5 comments
Labels

Comments

@edsiper
Copy link
Member

edsiper commented Mar 23, 2023

Bug Report

When an input plugin by configuration or flags set in the code uses coroutines and runs in a separate thread, there is a high chance it would crash. It seems to be related to the coroutines context handling inside the thread.

The easiest way to reproduce the issue is with this configuration (after #7060 gets merged in GIT master):

[SERVICE]
    Flush        0.5
    Daemon       Off
    Log_Level    info
    http_server  on
    http_listen  127.0.0.1
    http_port    2020

[INPUT]
    name  event_type
    type  logs
    interval_sec 0
    interval_nsec 500

[INPUT]
    name  event_type
    type  metrics
    interval_sec 0
    interval_nsec 500

[OUTPUT]
    Name  vivo_exporter
    Match *

the output:

[2023/03/23 19:39:21] [ info] [input:event_type:event_type.0] [OK] collector_time
[2023/03/23 19:39:21] [ info] [input:event_type:event_type.0] [OK] collector_time
[2023/03/23 19:39:21] [ info] [input:event_type:event_type.1] [OK] collector_time
[2023/03/23 19:39:21] [ info] [input:event_type:event_type.0] [OK] collector_time
[2023/03/23 19:39:21] [ info] [input:event_type:event_type.0] [OK] collector_time
[2023/03/23 19:39:21] [engine] caught signal (SIGSEGV)
*** stack smashing detected ***: terminated
[2023/03/23 19:39:21] [ info] [input:event_type:event_type.0] [OK] collector_time
Aborted (core dumped)
@nokute78
Copy link
Collaborator

Invalid handle was passed to co_switch and it caused crashing fluent-bit.

Disasemble/Backtrace

Following disasemble results points invalid reading a address 0x42b564e167fe6291 caused crash.
The address is pointed by rdi.

(gdb) disass
Dump of assembler code for function co_swap_function:
   0x0000555555f84ce0 <+0>:	mov    %rsp,(%rsi)
=> 0x0000555555f84ce3 <+3>:	mov    (%rdi),%rsp
   0x0000555555f84ce6 <+6>:	pop    %rax
(gdb) bt
#0  0x0000555555f84ce3 in co_swap_function ()
#1  0x0000555555f85ee9 in co_switch (handle=0x42b564e167fe6291)
    at /home/taka/git/fluent-bit/lib/monkey/deps/flb_libco/amd64.c:158
#2  0x000055555561b24a in input_pre_cb_collect () at /home/taka/git/fluent-bit/include/fluent-bit/flb_input.h:485
#3  0x0000555555f85ceb in co_init () at /home/taka/git/fluent-bit/lib/monkey/deps/flb_libco/amd64.c:117
#4  0x0000000000006210 in ?? ()

Registers

rdi is 0x42b564e167fe6291.
It is same value 0x0000555555f85ee9 in co_switch (handle=0x42b564e167fe6291) from backtrace.

(gdb) info register
rax            0x42b564e167fe6291  4806859096563016337
rbx            0x7fffaa00a950      140736045558096
rcx            0x0                 0
rdx            0x42b564e167fe6291  4806859096563016337
rsi            0x7fffaa00a950      140736045558096
rdi            0x42b564e167fe6291  4806859096563016337
rbp            0x7fffaa010b10      0x7fffaa010b10
rsp            0x7fffaa010ae8      0x7fffaa010ae8
r8             0x0                 0
r9             0x7fffaa00a950      140736045558096
r10            0x7fffec0004a0      140737152812192
r11            0x7fffec000090      140737152811152
r12            0x555555f84ce0      93825002917088
r13            0x0                 0
r14            0x0                 0
r15            0x0                 0
rip            0x555555f84ce3      0x555555f84ce3 <co_swap_function+3>
eflags         0x10206             [ PF IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0

Memory map

Following is a memory map of fluent-bit.
0x42b564e167fe6291 is out of range.

(gdb) info proc map
process 28061
Mapped address spaces:

          Start Addr           End Addr       Size     Offset  Perms  objfile
      0x555555554000     0x5555555cf000    0x7b000        0x0  r--p   /home/taka/git/fluent-bit/build/bin/fluent-bit
      0x5555555cf000     0x555555f89000   0x9ba000    0x7b000  r-xp   /home/taka/git/fluent-bit/build/bin/fluent-bit
      0x555555f89000     0x55555622c000   0x2a3000   0xa35000  r--p   /home/taka/git/fluent-bit/build/bin/fluent-bit
      0x55555622d000     0x55555623d000    0x10000   0xcd8000  r--p   /home/taka/git/fluent-bit/build/bin/fluent-bit
      0x55555623d000     0x55555625d000    0x20000   0xce8000  rw-p   /home/taka/git/fluent-bit/build/bin/fluent-bit
      0x55555625d000     0x5555562a2000    0x45000        0x0  rw-p   [heap]
      0x7fffa4000000     0x7fffaa013000  0x6013000        0x0  rw-p   
      0x7fffaa013000     0x7fffac000000  0x1fed000        0x0  ---p   
      0x7fffac000000     0x7fffb3ffd000  0x7ffd000        0x0  rw-p   
      0x7fffb3ffd000     0x7fffb4000000     0x3000        0x0  ---p   
      0x7fffb4000000     0x7fffbc000000  0x8000000        0x0  rw-p   
      0x7fffbc000000     0x7fffc3ffc000  0x7ffc000        0x0  rw-p   
      0x7fffc3ffc000     0x7fffc4000000     0x4000        0x0  ---p   
      0x7fffc4000000     0x7fffc7ffe000  0x3ffe000        0x0  rw-p   
      0x7fffc7ffe000     0x7fffc8000000     0x2000        0x0  ---p   
      0x7fffc8000000     0x7fffcc000000  0x4000000        0x0  rw-p   
      0x7fffcc000000     0x7fffcc021000    0x21000        0x0  rw-p   
      0x7fffcc021000     0x7fffd0000000  0x3fdf000        0x0  ---p   
      0x7fffd0000000     0x7fffd4000000  0x4000000        0x0  rw-p   
      0x7fffd4000000     0x7fffd5568000  0x1568000        0x0  rw-p   
      0x7fffd5568000     0x7fffd8000000  0x2a98000        0x0  ---p   
      0x7fffd8000000     0x7fffd8021000    0x21000        0x0  rw-p   
      0x7fffd8021000     0x7fffdc000000  0x3fdf000        0x0  ---p   
      0x7fffdc000000     0x7fffdc021000    0x21000        0x0  rw-p   
      0x7fffdc021000     0x7fffe0000000  0x3fdf000        0x0  ---p   
      0x7fffe0000000     0x7fffe0021000    0x21000        0x0  rw-p   
      0x7fffe0021000     0x7fffe4000000  0x3fdf000        0x0  ---p   
      0x7fffe57fb000     0x7fffe57fc000     0x1000        0x0  ---p   
      0x7fffe57fc000     0x7fffe5ffc000   0x800000        0x0  rwxp   
      0x7fffe5ffc000     0x7fffe5ffd000     0x1000        0x0  ---p   
      0x7fffe5ffd000     0x7fffe67fd000   0x800000        0x0  rwxp   
      0x7fffe67fd000     0x7fffe67fe000     0x1000        0x0  ---p   
      0x7fffe67fe000     0x7fffe6ffe000   0x800000        0x0  rwxp   
      0x7fffe6ffe000     0x7fffe6fff000     0x1000        0x0  ---p   
      0x7fffe6fff000     0x7fffe77ff000   0x800000        0x0  rwxp   
      0x7fffe77ff000     0x7fffe7800000     0x1000        0x0  ---p   
      0x7fffe7800000     0x7fffe8000000   0x800000        0x0  rwxp   
      0x7fffe8000000     0x7fffe8021000    0x21000        0x0  rw-p   
      0x7fffe8021000     0x7fffec000000  0x3fdf000        0x0  ---p   
      0x7fffec000000     0x7ffff0000000  0x4000000        0x0  rw-p   
      0x7ffff0000000     0x7ffff0783000   0x783000        0x0  rw-p   
      0x7ffff0783000     0x7ffff4000000  0x387d000        0x0  ---p   
      0x7ffff42bc000     0x7ffff42bd000     0x1000        0x0  ---p   
      0x7ffff42bd000     0x7ffff4abd000   0x800000        0x0  rwxp   
      0x7ffff4abd000     0x7ffff4abe000     0x1000        0x0  ---p   
      0x7ffff4abe000     0x7ffff52be000   0x800000        0x0  rwxp   
      0x7ffff52be000     0x7ffff52bf000     0x1000        0x0  ---p   
      0x7ffff52bf000     0x7ffff5abf000   0x800000        0x0  rwxp   
      0x7ffff5abf000     0x7ffff5ac0000     0x1000        0x0  ---p   
      0x7ffff5ac0000     0x7ffff62c0000   0x800000        0x0  rwxp   
      0x7ffff62c0000     0x7ffff62c1000     0x1000        0x0  ---p   
      0x7ffff62c1000     0x7ffff6ac1000   0x800000        0x0  rwxp   
      0x7ffff6ac1000     0x7ffff6ac2000     0x1000        0x0  ---p   
      0x7ffff6ac2000     0x7ffff72c2000   0x800000        0x0  rwxp   
      0x7ffff72c2000     0x7ffff72d1000     0xf000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.3.4
      0x7ffff72d1000     0x7ffff73b7000    0xe6000     0xf000  r-xp   /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.3.4
      0x7ffff73b7000     0x7ffff73f5000    0x3e000    0xf5000  r--p   /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.3.4
      0x7ffff73f5000     0x7ffff73f6000     0x1000   0x133000  ---p   /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.3.4
      0x7ffff73f6000     0x7ffff73f9000     0x3000   0x133000  r--p   /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.3.4
      0x7ffff73f9000     0x7ffff73ff000     0x6000   0x136000  rw-p   /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.3.4
      0x7ffff73ff000     0x7ffff7400000     0x1000        0x0  rw-p   
      0x7ffff7400000     0x7ffff7428000    0x28000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7ffff7428000     0x7ffff75bd000   0x195000    0x28000  r-xp   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7ffff75bd000     0x7ffff7615000    0x58000   0x1bd000  r--p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7ffff7615000     0x7ffff7619000     0x4000   0x214000  r--p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7ffff7619000     0x7ffff761b000     0x2000   0x218000  rw-p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7ffff761b000     0x7ffff7628000     0xd000        0x0  rw-p   
      0x7ffff7731000     0x7ffff773b000     0xa000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libzstd.so.1.4.8
      0x7ffff773b000     0x7ffff77ed000    0xb2000     0xa000  r-xp   /usr/lib/x86_64-linux-gnu/libzstd.so.1.4.8
      0x7ffff77ed000     0x7ffff77fe000    0x11000    0xbc000  r--p   /usr/lib/x86_64-linux-gnu/libzstd.so.1.4.8
      0x7ffff77fe000     0x7ffff77ff000     0x1000    0xcc000  r--p   /usr/lib/x86_64-linux-gnu/libzstd.so.1.4.8
      0x7ffff77ff000     0x7ffff7800000     0x1000    0xcd000  rw-p   /usr/lib/x86_64-linux-gnu/libzstd.so.1.4.8
      0x7ffff7800000     0x7ffff78b2000    0xb2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libcrypto.so.3
      0x7ffff78b2000     0x7ffff7b10000   0x25e000    0xb2000  r-xp   /usr/lib/x86_64-linux-gnu/libcrypto.so.3
      0x7ffff7b10000     0x7ffff7be2000    0xd2000   0x310000  r--p   /usr/lib/x86_64-linux-gnu/libcrypto.so.3
      0x7ffff7be2000     0x7ffff7c3d000    0x5b000   0x3e1000  r--p   /usr/lib/x86_64-linux-gnu/libcrypto.so.3
      0x7ffff7c3d000     0x7ffff7c40000     0x3000   0x43c000  rw-p   /usr/lib/x86_64-linux-gnu/libcrypto.so.3
      0x7ffff7c40000     0x7ffff7c43000     0x3000        0x0  rw-p   
      0x7ffff7c96000     0x7ffff7c98000     0x2000        0x0  rw-p   
      0x7ffff7c98000     0x7ffff7c9c000     0x4000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.32.1
      0x7ffff7c9c000     0x7ffff7cb2000    0x16000     0x4000  r-xp   /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.32.1
      0x7ffff7cb2000     0x7ffff7cbc000     0xa000    0x1a000  r--p   /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.32.1
      0x7ffff7cbc000     0x7ffff7cbd000     0x1000    0x23000  r--p   /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.32.1
      0x7ffff7cbd000     0x7ffff7cbe000     0x1000    0x24000  rw-p   /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.32.1
      0x7ffff7cbe000     0x7ffff7cc0000     0x2000        0x0  rw-p   
      0x7ffff7cc0000     0x7ffff7cc3000     0x3000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libcap.so.2.44
      0x7ffff7cc3000     0x7ffff7cc7000     0x4000     0x3000  r-xp   /usr/lib/x86_64-linux-gnu/libcap.so.2.44
      0x7ffff7cc7000     0x7ffff7cc9000     0x2000     0x7000  r--p   /usr/lib/x86_64-linux-gnu/libcap.so.2.44
      0x7ffff7cc9000     0x7ffff7cca000     0x1000     0x8000  r--p   /usr/lib/x86_64-linux-gnu/libcap.so.2.44
      0x7ffff7cca000     0x7ffff7ccb000     0x1000     0x9000  rw-p   /usr/lib/x86_64-linux-gnu/libcap.so.2.44
      0x7ffff7ccb000     0x7ffff7ccd000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.3
      0x7ffff7ccd000     0x7ffff7ce6000    0x19000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.3
      0x7ffff7ce6000     0x7ffff7ce8000     0x2000    0x1b000  r--p   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.3
      0x7ffff7ce8000     0x7ffff7ce9000     0x1000    0x1d000  ---p   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.3
      0x7ffff7ce9000     0x7ffff7cea000     0x1000    0x1d000  r--p   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.3
      0x7ffff7cea000     0x7ffff7ceb000     0x1000    0x1e000  rw-p   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.3
      0x7ffff7ceb000     0x7ffff7cee000     0x3000        0x0  r--p   /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5
      0x7ffff7cee000     0x7ffff7d09000    0x1b000     0x3000  r-xp   /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5
      0x7ffff7d09000     0x7ffff7d14000     0xb000    0x1e000  r--p   /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5
      0x7ffff7d14000     0x7ffff7d15000     0x1000    0x28000  r--p   /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5
      0x7ffff7d15000     0x7ffff7d16000     0x1000    0x29000  rw-p   /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5
      0x7ffff7d16000     0x7ffff7d18000     0x2000        0x0  rw-p   
      0x7ffff7d18000     0x7ffff7d1b000     0x3000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      0x7ffff7d1b000     0x7ffff7d32000    0x17000     0x3000  r-xp   /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      0x7ffff7d32000     0x7ffff7d36000     0x4000    0x1a000  r--p   /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      0x7ffff7d36000     0x7ffff7d37000     0x1000    0x1d000  r--p   /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      0x7ffff7d37000     0x7ffff7d38000     0x1000    0x1e000  rw-p   /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      0x7ffff7d38000     0x7ffff7d4b000    0x13000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libsystemd.so.0.32.0
      0x7ffff7d4b000     0x7ffff7dca000    0x7f000    0x13000  r-xp   /usr/lib/x86_64-linux-gnu/libsystemd.so.0.32.0
      0x7ffff7dca000     0x7ffff7df5000    0x2b000    0x92000  r--p   /usr/lib/x86_64-linux-gnu/libsystemd.so.0.32.0
      0x7ffff7df5000     0x7ffff7df6000     0x1000    0xbd000  ---p   /usr/lib/x86_64-linux-gnu/libsystemd.so.0.32.0
      0x7ffff7df6000     0x7ffff7dfd000     0x7000    0xbd000  r--p   /usr/lib/x86_64-linux-gnu/libsystemd.so.0.32.0
      0x7ffff7dfd000     0x7ffff7dfe000     0x1000    0xc4000  rw-p   /usr/lib/x86_64-linux-gnu/libsystemd.so.0.32.0
      0x7ffff7dfe000     0x7ffff7dff000     0x1000        0x0  rw-p   
      0x7ffff7dff000     0x7ffff7e01000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6
      0x7ffff7e01000     0x7ffff7e1a000    0x19000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6
      0x7ffff7e1a000     0x7ffff7e1e000     0x4000    0x1b000  r--p   /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6
      0x7ffff7e1e000     0x7ffff7e1f000     0x1000    0x1e000  r--p   /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6
      0x7ffff7e1f000     0x7ffff7e20000     0x1000    0x1f000  rw-p   /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6
      0x7ffff7e20000     0x7ffff7e3e000    0x1e000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libssl.so.3
      0x7ffff7e3e000     0x7ffff7e99000    0x5b000    0x1e000  r-xp   /usr/lib/x86_64-linux-gnu/libssl.so.3
      0x7ffff7e99000     0x7ffff7eb6000    0x1d000    0x79000  r--p   /usr/lib/x86_64-linux-gnu/libssl.so.3
      0x7ffff7eb6000     0x7ffff7ec0000     0xa000    0x95000  r--p   /usr/lib/x86_64-linux-gnu/libssl.so.3
      0x7ffff7ec0000     0x7ffff7ec4000     0x4000    0x9f000  rw-p   /usr/lib/x86_64-linux-gnu/libssl.so.3
      0x7ffff7ec4000     0x7ffff7ed2000     0xe000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libm.so.6
      0x7ffff7ed2000     0x7ffff7f4e000    0x7c000     0xe000  r-xp   /usr/lib/x86_64-linux-gnu/libm.so.6
      0x7ffff7f4e000     0x7ffff7fa9000    0x5b000    0x8a000  r--p   /usr/lib/x86_64-linux-gnu/libm.so.6
      0x7ffff7fa9000     0x7ffff7faa000     0x1000    0xe4000  r--p   /usr/lib/x86_64-linux-gnu/libm.so.6
      0x7ffff7faa000     0x7ffff7fab000     0x1000    0xe5000  rw-p   /usr/lib/x86_64-linux-gnu/libm.so.6
      0x7ffff7fbb000     0x7ffff7fbd000     0x2000        0x0  rw-p   
      0x7ffff7fbd000     0x7ffff7fc1000     0x4000        0x0  r--p   [vvar]
      0x7ffff7fc1000     0x7ffff7fc3000     0x2000        0x0  r-xp   [vdso]
      0x7ffff7fc3000     0x7ffff7fc5000     0x2000        0x0  r--p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffff7fc5000     0x7ffff7fef000    0x2a000     0x2000  r-xp   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffff7fef000     0x7ffff7ffa000     0xb000    0x2c000  r--p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffff7ffa000     0x7ffff7ffb000     0x1000        0x0  rw-p   
      0x7ffff7ffb000     0x7ffff7ffd000     0x2000    0x37000  r--p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffff7ffd000     0x7ffff7fff000     0x2000    0x39000  rw-p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffffffde000     0x7ffffffff000    0x21000        0x0  rwxp   [stack]
  0xffffffffff600000 0xffffffffff601000     0x1000        0x0  --xp   [vsyscall]

@nokute78
Copy link
Collaborator

nokute78 commented Apr 1, 2023

The root cause is a race of libco_in_param.
I added a mutex and it doesn't cause SIGSEGV.
But it affected fluent-bit didn't exit when user sents Ctrl+C(SIGINT)...

https://github.com/nokute78/fluent-bit/tree/fix_7061

nokute78 added a commit to nokute78/fluent-bit that referenced this issue Apr 1, 2023
Signed-off-by: Takahiro Yamashita <nokute78@gmail.com>
nokute78 added a commit to nokute78/fluent-bit that referenced this issue Apr 1, 2023
Signed-off-by: Takahiro Yamashita <nokute78@gmail.com>
@nokute78
Copy link
Collaborator

nokute78 commented Apr 1, 2023

Oops following commit is a wrong. Please ignore it...
https://github.com/nokute78/fluent-bit/tree/fix_7061

@nokute78
Copy link
Collaborator

nokute78 commented Apr 1, 2023

I sent a patch #7102
It uses pthread_key to fix this issue, but many valgrind errors were reported.

@edsiper
Copy link
Member Author

edsiper commented Jun 14, 2023

fixed in #7102

@edsiper edsiper closed this as completed Jun 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants