Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FluentBit caught SIGSEGV when threaded is enabled in the "forward" input #6865

Closed
panaji opened this issue Feb 14, 2023 · 12 comments
Closed

FluentBit caught SIGSEGV when threaded is enabled in the "forward" input #6865

panaji opened this issue Feb 14, 2023 · 12 comments
Labels
bug fixed Stale

Comments

@panaji
Copy link

panaji commented Feb 14, 2023

Bug Report

After upgrading fluentbit from v1.9.10 to v2.0.9, we start seeing this SIGSEGV error. Our logging pipeline uses the forwarder aggregator pattern. We notice this problem in the aggregator component when we turn on the "threaded" parameter in the "forward" input. We get this SIGSEGV error once/twice an hour.

Describe the bug

To Reproduce

  • Steps to reproduce the problem:
    Set fluentbit with forwarder and aggregator pattern. In the aggregator, turn on the "threaded" parameter in the "forward" input section. You might need to wait for a while for the error to show up.
[INPUT]
    Name     forward
    Alias    input.forward
    Listen   0.0.0.0
    Port     24224
    Threaded On
  • Example log message if applicable:
NAME                                    READY   STATUS    RESTARTS        AGE
genevabit-aggregator-776f9cdf7f-8wsw2   2/2     Running   124 (81m ago)   2d20h

[2023/02/13 16:31:49] [engine] caught signal (SIGSEGV)
#0  0x55c80326bf40      in  __mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:141
#1  0x55c80326bf77      in  mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:147
#2  0x55c80326c27d      in  flb_bucket_queue_delete_min() at include/fluent-bit/flb_bucket_queue.h:113
#3  0x55c80326c2ca      in  flb_bucket_queue_pop_min() at include/fluent-bit/flb_bucket_queue.h:122
#4  0x55c803271b31      in  flb_engine_start() at src/flb_engine.c:851
#5  0x55c803218964      in  flb_lib_worker() at src/flb_lib.c:629
#6  0x7f431b427ea6      in  start_thread() at reate.c:477
#7  0x7f431acdba2e      in  ???() at sysv/linux/x86_64/clone.S:95
#8  0xffffffffffffffff  in  ???() at ???:0

Expected behavior
Don't see this error.

Screenshots

Your Environment

  • Version used: v2.0.9
  • Configuration:
  • Environment name and version (e.g. Kubernetes? What version?): k8s 1.24.9
  • Server type and version:
  • Operating System and version:
  • Filters and plugins:

Additional context
@patrick-stephens
Copy link
Contributor

Possibly related to #6821

@nokute78
Copy link
Collaborator

#6492 seems to be same issue.

@nokute78
Copy link
Collaborator

nokute78 commented Feb 18, 2023 •
edited
Loading

I tried to use thread sanitizer and some races were reported.

cmake .. -DSANITIZE_THREAD=on && make

Fluent-bit-1

[INPUT]
    Name forward
    Threaded on

[OUTPUT]
    Name null

Fluent-bit-2

fluent-bit -i dummy -p rate=100000 -o forward

==================
WARNING: ThreadSanitizer: data race (pid=47741)
  Write of size 8 at 0x7b3800010048 by thread T3:
    #0 __mk_list_add /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:60 (fluent-bit+0x144c66)
    #1 mk_list_add /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:65 (fluent-bit+0x144cbc)
    #2 flb_downstream_conn_get /home/taka/git/fluent-bit/src/flb_downstream.c:299 (fluent-bit+0x14a6f4)
    #3 in_fw_collect /home/taka/git/fluent-bit/plugins/in_forward/fw.c:130 (fluent-bit+0x49a274)
    #4 input_collector_fd /home/taka/git/fluent-bit/src/flb_input_thread.c:168 (fluent-bit+0xe1395)
    #5 engine_handle_event /home/taka/git/fluent-bit/src/flb_input_thread.c:183 (fluent-bit+0xe2111)
    #6 input_thread /home/taka/git/fluent-bit/src/flb_input_thread.c:384 (fluent-bit+0xe2111)
    #7 step_callback /home/taka/git/fluent-bit/src/flb_worker.c:43 (fluent-bit+0x158b28)

  Previous read of size 8 at 0x7b3800010048 by thread T1:
    #0 flb_downstream_conn_timeouts /home/taka/git/fluent-bit/src/flb_downstream.c:418 (fluent-bit+0x14abd6)
    #1 cb_engine_sched_timer /home/taka/git/fluent-bit/src/flb_engine.c:172 (fluent-bit+0x124a94)
    #2 flb_sched_event_handler /home/taka/git/fluent-bit/src/flb_scheduler.c:428 (fluent-bit+0x13ad70)
    #3 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:945 (fluent-bit+0x127254)
    #4 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

  Location is heap block of size 224 at 0x7b380000ff80 allocated by thread T3:
    #0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:672 (libtsan.so.0+0x31edc)
    #1 flb_calloc /home/taka/git/fluent-bit/include/fluent-bit/flb_mem.h:95 (fluent-bit+0x144e92)
    #2 flb_downstream_create /home/taka/git/fluent-bit/src/flb_downstream.c:152 (fluent-bit+0x14a256)
    #3 in_fw_init /home/taka/git/fluent-bit/plugins/in_forward/fw.c:198 (fluent-bit+0x49a83f)
    #4 input_thread /home/taka/git/fluent-bit/src/flb_input_thread.c:339 (fluent-bit+0xe1d1a)
    #5 step_callback /home/taka/git/fluent-bit/src/flb_worker.c:43 (fluent-bit+0x158b28)

  Thread T3 'flb-in-forward.' (tid=47745, running) created by thread T1 at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (fluent-bit+0xc2634c)
    #2 flb_tp_thread_start /home/taka/git/fluent-bit/src/flb_thread_pool.c:123 (fluent-bit+0x17135d)
    #3 flb_input_thread_instance_init /home/taka/git/fluent-bit/src/flb_input_thread.c:543 (fluent-bit+0xe2cf6)
    #4 flb_input_instance_init /home/taka/git/fluent-bit/src/flb_input.c:1130 (fluent-bit+0xd83d0)
    #5 flb_input_init_all /home/taka/git/fluent-bit/src/flb_input.c:1217 (fluent-bit+0xd88a3)
    #6 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:725 (fluent-bit+0x126424)
    #7 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

  Thread T1 'flb-pipeline' (tid=47743, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (fluent-bit+0xc2634c)
    #2 flb_main /home/taka/git/fluent-bit/src/fluent-bit.c:1231 (fluent-bit+0x881f1)
    #3 main /home/taka/git/fluent-bit/src/fluent-bit.c:1257 (fluent-bit+0x8834e)

SUMMARY: ThreadSanitizer: data race /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:60 in __mk_list_add
==================
==================
WARNING: ThreadSanitizer: data race (pid=47741)
  Read of size 8 at 0x7b1400005040 by thread T1:
    #0 mk_list_del /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:147 (fluent-bit+0x49afb0)
    #1 fw_conn_del /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:179 (fluent-bit+0x4a0844)
    #2 fw_conn_event /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:101 (fluent-bit+0x4a02ea)
    #3 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:961 (fluent-bit+0x1273d2)
    #4 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

  Previous write of size 8 at 0x7b1400005040 by thread T3:
    #0 __mk_list_add /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:58 (fluent-bit+0x49ae9f)
    #1 mk_list_add /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:65 (fluent-bit+0x49af28)
    #2 fw_conn_add /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:166 (fluent-bit+0x4a07e2)
    #3 in_fw_collect /home/taka/git/fluent-bit/plugins/in_forward/fw.c:146 (fluent-bit+0x49a45e)
    #4 input_collector_fd /home/taka/git/fluent-bit/src/flb_input_thread.c:168 (fluent-bit+0xe1395)
    #5 engine_handle_event /home/taka/git/fluent-bit/src/flb_input_thread.c:183 (fluent-bit+0xe2111)
    #6 input_thread /home/taka/git/fluent-bit/src/flb_input_thread.c:384 (fluent-bit+0xe2111)
    #7 step_callback /home/taka/git/fluent-bit/src/flb_worker.c:43 (fluent-bit+0x158b28)

  Location is heap block of size 72 at 0x7b1400005000 allocated by thread T3:
    #0 malloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:655 (libtsan.so.0+0x31c57)
    #1 flb_malloc /home/taka/git/fluent-bit/include/fluent-bit/flb_mem.h:80 (fluent-bit+0x49b068)
    #2 fw_conn_add /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:120 (fluent-bit+0x4a0463)
    #3 in_fw_collect /home/taka/git/fluent-bit/plugins/in_forward/fw.c:146 (fluent-bit+0x49a45e)
    #4 input_collector_fd /home/taka/git/fluent-bit/src/flb_input_thread.c:168 (fluent-bit+0xe1395)
    #5 engine_handle_event /home/taka/git/fluent-bit/src/flb_input_thread.c:183 (fluent-bit+0xe2111)
    #6 input_thread /home/taka/git/fluent-bit/src/flb_input_thread.c:384 (fluent-bit+0xe2111)
    #7 step_callback /home/taka/git/fluent-bit/src/flb_worker.c:43 (fluent-bit+0x158b28)

  Thread T1 'flb-pipeline' (tid=47743, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (fluent-bit+0xc2634c)
    #2 flb_main /home/taka/git/fluent-bit/src/fluent-bit.c:1231 (fluent-bit+0x881f1)
    #3 main /home/taka/git/fluent-bit/src/fluent-bit.c:1257 (fluent-bit+0x8834e)

  Thread T3 'flb-in-forward.' (tid=47745, running) created by thread T1 at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (fluent-bit+0xc2634c)
    #2 flb_tp_thread_start /home/taka/git/fluent-bit/src/flb_thread_pool.c:123 (fluent-bit+0x17135d)
    #3 flb_input_thread_instance_init /home/taka/git/fluent-bit/src/flb_input_thread.c:543 (fluent-bit+0xe2cf6)
    #4 flb_input_instance_init /home/taka/git/fluent-bit/src/flb_input.c:1130 (fluent-bit+0xd83d0)
    #5 flb_input_init_all /home/taka/git/fluent-bit/src/flb_input.c:1217 (fluent-bit+0xd88a3)
    #6 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:725 (fluent-bit+0x126424)
    #7 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

SUMMARY: ThreadSanitizer: data race /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:147 in mk_list_del
==================
==================
WARNING: ThreadSanitizer: data race (pid=47741)
  Read of size 8 at 0x7b1400005038 by thread T1:
    #0 mk_list_del /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:147 (fluent-bit+0x49afc4)
    #1 fw_conn_del /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:179 (fluent-bit+0x4a0844)
    #2 fw_conn_event /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:101 (fluent-bit+0x4a02ea)
    #3 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:961 (fluent-bit+0x1273d2)
    #4 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

  Previous write of size 8 at 0x7b1400005038 by thread T3:
    #0 __mk_list_add /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:59 (fluent-bit+0x49aeb7)
    #1 mk_list_add /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:65 (fluent-bit+0x49af28)
    #2 fw_conn_add /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:166 (fluent-bit+0x4a07e2)
    #3 in_fw_collect /home/taka/git/fluent-bit/plugins/in_forward/fw.c:146 (fluent-bit+0x49a45e)
    #4 input_collector_fd /home/taka/git/fluent-bit/src/flb_input_thread.c:168 (fluent-bit+0xe1395)
    #5 engine_handle_event /home/taka/git/fluent-bit/src/flb_input_thread.c:183 (fluent-bit+0xe2111)
    #6 input_thread /home/taka/git/fluent-bit/src/flb_input_thread.c:384 (fluent-bit+0xe2111)
    #7 step_callback /home/taka/git/fluent-bit/src/flb_worker.c:43 (fluent-bit+0x158b28)

  Location is heap block of size 72 at 0x7b1400005000 allocated by thread T3:
    #0 malloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:655 (libtsan.so.0+0x31c57)
    #1 flb_malloc /home/taka/git/fluent-bit/include/fluent-bit/flb_mem.h:80 (fluent-bit+0x49b068)
    #2 fw_conn_add /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:120 (fluent-bit+0x4a0463)
    #3 in_fw_collect /home/taka/git/fluent-bit/plugins/in_forward/fw.c:146 (fluent-bit+0x49a45e)
    #4 input_collector_fd /home/taka/git/fluent-bit/src/flb_input_thread.c:168 (fluent-bit+0xe1395)
    #5 engine_handle_event /home/taka/git/fluent-bit/src/flb_input_thread.c:183 (fluent-bit+0xe2111)
    #6 input_thread /home/taka/git/fluent-bit/src/flb_input_thread.c:384 (fluent-bit+0xe2111)
    #7 step_callback /home/taka/git/fluent-bit/src/flb_worker.c:43 (fluent-bit+0x158b28)

  Thread T1 'flb-pipeline' (tid=47743, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (fluent-bit+0xc2634c)
    #2 flb_main /home/taka/git/fluent-bit/src/fluent-bit.c:1231 (fluent-bit+0x881f1)
    #3 main /home/taka/git/fluent-bit/src/fluent-bit.c:1257 (fluent-bit+0x8834e)

  Thread T3 'flb-in-forward.' (tid=47745, running) created by thread T1 at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (fluent-bit+0xc2634c)
    #2 flb_tp_thread_start /home/taka/git/fluent-bit/src/flb_thread_pool.c:123 (fluent-bit+0x17135d)
    #3 flb_input_thread_instance_init /home/taka/git/fluent-bit/src/flb_input_thread.c:543 (fluent-bit+0xe2cf6)
    #4 flb_input_instance_init /home/taka/git/fluent-bit/src/flb_input.c:1130 (fluent-bit+0xd83d0)
    #5 flb_input_init_all /home/taka/git/fluent-bit/src/flb_input.c:1217 (fluent-bit+0xd88a3)
    #6 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:725 (fluent-bit+0x126424)
    #7 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

SUMMARY: ThreadSanitizer: data race /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:147 in mk_list_del
==================
==================
WARNING: ThreadSanitizer: data race (pid=47741)
  Write of size 8 at 0x7b1c00003858 by thread T1:
    #0 __mk_list_del /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:141 (fluent-bit+0x49af5c)
    #1 mk_list_del /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:147 (fluent-bit+0x49afd6)
    #2 fw_conn_del /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:179 (fluent-bit+0x4a0844)
    #3 fw_conn_event /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:101 (fluent-bit+0x4a02ea)
    #4 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:961 (fluent-bit+0x1273d2)
    #5 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

  Previous write of size 8 at 0x7b1c00003858 by thread T3:
    #0 __mk_list_add /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:60 (fluent-bit+0x49aed2)
    #1 mk_list_add /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:65 (fluent-bit+0x49af28)
    #2 fw_conn_add /home/taka/git/fluent-bit/plugins/in_forward/fw_conn.c:166 (fluent-bit+0x4a07e2)
    #3 in_fw_collect /home/taka/git/fluent-bit/plugins/in_forward/fw.c:146 (fluent-bit+0x49a45e)
    #4 input_collector_fd /home/taka/git/fluent-bit/src/flb_input_thread.c:168 (fluent-bit+0xe1395)
    #5 engine_handle_event /home/taka/git/fluent-bit/src/flb_input_thread.c:183 (fluent-bit+0xe2111)
    #6 input_thread /home/taka/git/fluent-bit/src/flb_input_thread.c:384 (fluent-bit+0xe2111)
    #7 step_callback /home/taka/git/fluent-bit/src/flb_worker.c:43 (fluent-bit+0x158b28)

  Location is heap block of size 112 at 0x7b1c00003800 allocated by thread T3:
    #0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:672 (libtsan.so.0+0x31edc)
    #1 flb_calloc /home/taka/git/fluent-bit/include/fluent-bit/flb_mem.h:95 (fluent-bit+0x4a8e12)
    #2 fw_config_init /home/taka/git/fluent-bit/plugins/in_forward/fw_config.c:36 (fluent-bit+0x4adae5)
    #3 in_fw_init /home/taka/git/fluent-bit/plugins/in_forward/fw.c:166 (fluent-bit+0x49a4c0)
    #4 input_thread /home/taka/git/fluent-bit/src/flb_input_thread.c:339 (fluent-bit+0xe1d1a)
    #5 step_callback /home/taka/git/fluent-bit/src/flb_worker.c:43 (fluent-bit+0x158b28)

  Thread T1 'flb-pipeline' (tid=47743, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (fluent-bit+0xc2634c)
    #2 flb_main /home/taka/git/fluent-bit/src/fluent-bit.c:1231 (fluent-bit+0x881f1)
    #3 main /home/taka/git/fluent-bit/src/fluent-bit.c:1257 (fluent-bit+0x8834e)

  Thread T3 'flb-in-forward.' (tid=47745, running) created by thread T1 at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (fluent-bit+0xc2634c)
    #2 flb_tp_thread_start /home/taka/git/fluent-bit/src/flb_thread_pool.c:123 (fluent-bit+0x17135d)
    #3 flb_input_thread_instance_init /home/taka/git/fluent-bit/src/flb_input_thread.c:543 (fluent-bit+0xe2cf6)
    #4 flb_input_instance_init /home/taka/git/fluent-bit/src/flb_input.c:1130 (fluent-bit+0xd83d0)
    #5 flb_input_init_all /home/taka/git/fluent-bit/src/flb_input.c:1217 (fluent-bit+0xd88a3)
    #6 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:725 (fluent-bit+0x126424)
    #7 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

SUMMARY: ThreadSanitizer: data race /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:141 in __mk_list_del
==================

@nokute78
Copy link
Collaborator

This is the race log and backtrace log of this issue.
I think it is a data race between flb_downstream_conn_get and flb_downstream_conn_timeouts.

s$ ../bin/fluent-bit -c 6865/a.conf 
Fluent Bit v2.1.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/02/18 10:18:07] [ info] [fluent bit] version=2.1.0, commit=e4c982757d, pid=48399
[2023/02/18 10:18:07] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/02/18 10:18:07] [ info] [cmetrics] version=0.5.8
[2023/02/18 10:18:07] [ info] [ctraces ] version=0.3.0
[2023/02/18 10:18:07] [ info] [input:forward:forward.0] initializing
[2023/02/18 10:18:07] [ info] [input:forward:forward.0] storage_strategy='memory' (memory only)
[2023/02/18 10:18:07] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2023/02/18 10:18:07] [ info] [input:forward:forward.0] thread instance initialized
[2023/02/18 10:18:07] [ info] [output:null:null.0] worker #0 started
[2023/02/18 10:18:07] [ info] [sp] stream processor started
==================
WARNING: ThreadSanitizer: data race (pid=48399)
  Write of size 8 at 0x7b3800010048 by thread T3:
    #0 __mk_list_add /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:60 (fluent-bit+0x144c66)
    #1 mk_list_add /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:65 (fluent-bit+0x144cbc)
    #2 flb_downstream_conn_get /home/taka/git/fluent-bit/src/flb_downstream.c:299 (fluent-bit+0x14a6f4)
    #3 in_fw_collect /home/taka/git/fluent-bit/plugins/in_forward/fw.c:130 (fluent-bit+0x49a274)
    #4 input_collector_fd /home/taka/git/fluent-bit/src/flb_input_thread.c:168 (fluent-bit+0xe1395)
    #5 engine_handle_event /home/taka/git/fluent-bit/src/flb_input_thread.c:183 (fluent-bit+0xe2111)
    #6 input_thread /home/taka/git/fluent-bit/src/flb_input_thread.c:384 (fluent-bit+0xe2111)
    #7 step_callback /home/taka/git/fluent-bit/src/flb_worker.c:43 (fluent-bit+0x158b28)

  Previous read of size 8 at 0x7b3800010048 by thread T1:
    #0 flb_downstream_conn_timeouts /home/taka/git/fluent-bit/src/flb_downstream.c:418 (fluent-bit+0x14abd6)
    #1 cb_engine_sched_timer /home/taka/git/fluent-bit/src/flb_engine.c:172 (fluent-bit+0x124a94)
    #2 flb_sched_event_handler /home/taka/git/fluent-bit/src/flb_scheduler.c:428 (fluent-bit+0x13ad70)
    #3 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:945 (fluent-bit+0x127254)
    #4 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

  Location is heap block of size 224 at 0x7b380000ff80 allocated by thread T3:
    #0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:672 (libtsan.so.0+0x31edc)
    #1 flb_calloc /home/taka/git/fluent-bit/include/fluent-bit/flb_mem.h:95 (fluent-bit+0x144e92)
    #2 flb_downstream_create /home/taka/git/fluent-bit/src/flb_downstream.c:152 (fluent-bit+0x14a256)
    #3 in_fw_init /home/taka/git/fluent-bit/plugins/in_forward/fw.c:198 (fluent-bit+0x49a83f)
    #4 input_thread /home/taka/git/fluent-bit/src/flb_input_thread.c:339 (fluent-bit+0xe1d1a)
    #5 step_callback /home/taka/git/fluent-bit/src/flb_worker.c:43 (fluent-bit+0x158b28)

  Thread T3 'flb-in-forward.' (tid=48403, running) created by thread T1 at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (fluent-bit+0xc2634c)
    #2 flb_tp_thread_start /home/taka/git/fluent-bit/src/flb_thread_pool.c:123 (fluent-bit+0x17135d)
    #3 flb_input_thread_instance_init /home/taka/git/fluent-bit/src/flb_input_thread.c:543 (fluent-bit+0xe2cf6)
    #4 flb_input_instance_init /home/taka/git/fluent-bit/src/flb_input.c:1130 (fluent-bit+0xd83d0)
    #5 flb_input_init_all /home/taka/git/fluent-bit/src/flb_input.c:1217 (fluent-bit+0xd88a3)
    #6 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:725 (fluent-bit+0x126424)
    #7 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

  Thread T1 'flb-pipeline' (tid=48401, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (fluent-bit+0xc2634c)
    #2 flb_main /home/taka/git/fluent-bit/src/fluent-bit.c:1231 (fluent-bit+0x881f1)
    #3 main /home/taka/git/fluent-bit/src/fluent-bit.c:1257 (fluent-bit+0x8834e)

SUMMARY: ThreadSanitizer: data race /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:60 in __mk_list_add
==================
==================
WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=48399)
    #0 free ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:711 (libtsan.so.0+0x37ab8)
    #1 tzset_internal time/tzset.c:401 (libc.so.6+0xdac78)
    #2 flb_signal_handler /home/taka/git/fluent-bit/src/fluent-bit.c:549 (fluent-bit+0x85c9d)
    #3 CallUserSignalHandler ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1924 (libtsan.so.0+0x2d883)
    #4 mk_list_del /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:147 (fluent-bit+0x11ee64)
    #5 flb_bucket_queue_delete_min /home/taka/git/fluent-bit/include/fluent-bit/flb_bucket_queue.h:113 (fluent-bit+0x11f409)
    #6 flb_bucket_queue_pop_min /home/taka/git/fluent-bit/include/fluent-bit/flb_bucket_queue.h:122 (fluent-bit+0x11f48c)
    #7 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:859 (fluent-bit+0x12766d)
    #8 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

SUMMARY: ThreadSanitizer: signal-unsafe call inside of a signal time/tzset.c:401 in tzset_internal
==================
==================
WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=48399)
    #0 malloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:655 (libtsan.so.0+0x31c57)
    #1 __GI___strdup string/strdup.c:42 (libc.so.6+0xa860e)
    #2 flb_signal_handler /home/taka/git/fluent-bit/src/fluent-bit.c:549 (fluent-bit+0x85c9d)
    #3 CallUserSignalHandler ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1924 (libtsan.so.0+0x2d883)
    #4 mk_list_del /home/taka/git/fluent-bit/lib/monkey/include/monkey/mk_core/mk_list.h:147 (fluent-bit+0x11ee64)
    #5 flb_bucket_queue_delete_min /home/taka/git/fluent-bit/include/fluent-bit/flb_bucket_queue.h:113 (fluent-bit+0x11f409)
    #6 flb_bucket_queue_pop_min /home/taka/git/fluent-bit/include/fluent-bit/flb_bucket_queue.h:122 (fluent-bit+0x11f48c)
    #7 flb_engine_start /home/taka/git/fluent-bit/src/flb_engine.c:859 (fluent-bit+0x12766d)
    #8 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:629 (fluent-bit+0xa1ced)

SUMMARY: ThreadSanitizer: signal-unsafe call inside of a signal string/strdup.c:42 in __GI___strdup
==================
[2023/02/18 10:18:11] [engine] caught signal (SIGSEGV)
#0  0x7fa8cd785bc9      in  _Z22rtl_generic_sighandlerbiPN11__sanitizer19__sanitizer_siginfoEPv() at er/tsan/tsan_interceptors_posix.cpp:2016
#1  0x7fa8cd785bc9      in  rtl_sighandler() at er/tsan/tsan_interceptors_posix.cpp:2037
#2  0x7fa8cce9851f      in  ???() at ysv/linux/x86_64/libc_sigaction.c:0
#3  0x55f30c207df3      in  __mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:141
#4  0x55f30c207e64      in  mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:147
#5  0x55f30c208409      in  flb_bucket_queue_delete_min() at include/fluent-bit/flb_bucket_queue.h:113
#6  0x55f30c20848c      in  flb_bucket_queue_pop_min() at include/fluent-bit/flb_bucket_queue.h:122
#7  0x55f30c21066d      in  flb_engine_start() at src/flb_engine.c:859
#8  0x55f30c18aced      in  flb_lib_worker() at src/flb_lib.c:629
#9  0x7fa8cd7845ff      in  __tsan_thread_start_func() at er/tsan/tsan_interceptors_posix.cpp:959
#10 0x7fa8cceeab42      in  start_thread() at /sysdeps/unix/sysv/linux/rseq-internal.h:442
#11 0x7fa8ccf7c9ff      in  ???() at /x86_64/clone3.S:81
#12 0xffffffffffffffff  in  ???() at ???:0
Aborted (core dumped)

@nokute78 nokute78 mentioned this issue Feb 19, 2023
Closed
3 tasks
@leonardo-albertovich
Copy link
Collaborator

Hi @panaji, I am in the process of testing the patch that should correct this issue and would really appreciate it if you could give it a try, the branch URL is https://github.com/fluent/fluent-bit/tree/leonardo-master-downstream-pr-6882-rework

Thank you very much.

@panaji
Copy link
Author

panaji commented Feb 23, 2023

@leonardo-albertovich , will do. Please ping me if i don't reply by Mon.

@nokute78
Copy link
Collaborator

Note: #6906 was merged.

@leonardo-albertovich
Copy link
Collaborator

Hi @panaji, were you able to run those tests?

@panaji
Copy link
Author

panaji commented Feb 28, 2023

@leonardo-albertovich , yes ... it's been running since last night without SIGSEGV, so i think we are good.

@leonardo-albertovich
Copy link
Collaborator

Awesome!

@github-actions
Copy link
Contributor

github-actions bot commented Jun 2, 2023

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

@github-actions github-actions bot added the Stale label Jun 2, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jun 7, 2023

This issue was closed because it has been stalled for 5 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jun 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug fixed Stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

downstream: add mutex for busy_queue nokute78/fluent-bit
5 participants
@leonardo-albertovich @lecaros @patrick-stephens @nokute78 @panaji