filter_grep: sporadic unit test failures

[rittneje]

flb-rt-filter_grep "Test invalid" fails sporadically with SIGSEGV. The logs do not make it clear what specifically is causing the failure.

Test invalid... [2021/07/24 17:44:25] [ info] [engine] started (pid=18409)
[2021/07/24 17:44:25] [ info] [storage] version=1.1.1, initializing...
[2021/07/24 17:44:25] [ info] [storage] in-memory
[2021/07/24 17:44:25] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/07/24 17:44:25] [ info] [cmetrics] version=0.1.5
Error: 7/24 17:44:25] [error] [filter:grep:grep.0] invalid regex, expected field and regular expression
Error: 7/24 17:44:25] [error] [lib] backend failed
Error: 7/24 17:44:25] [error] Failed initialize filter grep.0
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Error: 7/24 17:44:25] [error] [lib] cannot push data, engine is not running
Test interrupted by SIGSEGV.
FAILED: 1 of 3 unit tests has failed.

[rittneje]

Another test run similarly failed with SIGABRT. It could be the same underlying issue.

double free or corruption (fasttop)
Test interrupted by SIGABRT.

[rittneje]

I have not been able to reproduce this failure locally. Running with Valgrind does detect a leak, but not a double free.

==7480== 320 bytes in 1 blocks are possibly lost in loss record 1 of 1
==7480== at 0x4C33B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7480== by 0x4013646: allocate_dtv (dl-tls.c:286)
==7480== by 0x4013646: dl_allocate_tls (dl-tls.c:530)
==7480== by 0x4E46227: allocate_stack (allocatestack.c:627)
==7480== by 0x4E46227: pthread_create@@GLIBC_2.2.5 (pthread_create.c:644)
==7480== by 0x84184C: mk_utils_worker_spawn (mk_utils.c:284)
==7480== by 0x326E63: flb_start (flb_lib.c:659)
==7480== by 0x323A35: flb_test_filter_grep_regex (filter_grep.c:43)
==7480== by 0x320B40: test_do_run (acutest.h:1007)
==7480== by 0x320FBE: test_run_ (acutest.h:1178)
==7480== by 0x323108: main (acutest.h:1700)
==7480==

[nokute78]

Note: Thread sanitizer detected data racing.

==================
WARNING: ThreadSanitizer: data race (pid=21267)
  Write of size 4 at 0x7b0800000020 by thread T1:
    #0 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:633 (flb-rt-filter_grep+0x903a0)

  Previous write of size 4 at 0x7b0800000020 by main thread:
    #0 flb_start /home/taka/git/fluent-bit/src/flb_lib.c:682 (flb-rt-filter_grep+0x9068c)
    #1 flb_test_filter_grep_invalid /home/taka/git/fluent-bit/tests/runtime/filter_grep.c:133 (flb-rt-filter_grep+0x8e084)
    #2 test_do_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1007 (flb-rt-filter_grep+0x8b315)
    #3 test_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1103 (flb-rt-filter_grep+0x8b699)
    #4 main /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1700 (flb-rt-filter_grep+0x8d0f3)

  Location is heap block of size 32 at 0x7b0800000020 allocated by main thread:
    #0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:668 (libtsan.so.0+0x305aa)
    #1 flb_calloc /home/taka/git/fluent-bit/include/fluent-bit/flb_mem.h:78 (flb-rt-filter_grep+0x8e3e0)
    #2 flb_create /home/taka/git/fluent-bit/src/flb_lib.c:146 (flb-rt-filter_grep+0x8e8ac)
    #3 flb_test_filter_grep_invalid /home/taka/git/fluent-bit/tests/runtime/filter_grep.c:114 (flb-rt-filter_grep+0x8de0e)
    #4 test_do_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1007 (flb-rt-filter_grep+0x8b315)
    #5 test_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1103 (flb-rt-filter_grep+0x8b699)
    #6 main /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1700 (flb-rt-filter_grep+0x8d0f3)

  Thread T1 'flb-pipeline' (tid=21270, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (flb-rt-filter_grep+0x68429a)
    #2 flb_test_filter_grep_invalid /home/taka/git/fluent-bit/tests/runtime/filter_grep.c:133 (flb-rt-filter_grep+0x8e084)
    #3 test_do_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1007 (flb-rt-filter_grep+0x8b315)
    #4 test_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1103 (flb-rt-filter_grep+0x8b699)
    #5 main /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1700 (flb-rt-filter_grep+0x8d0f3)

SUMMARY: ThreadSanitizer: data race /home/taka/git/fluent-bit/src/flb_lib.c:633 in flb_lib_worker
==================
==================
WARNING: ThreadSanitizer: data race (pid=21267)
  Read of size 4 at 0x7ba400000038 by main thread:
    #0 flb_destroy /home/taka/git/fluent-bit/src/flb_lib.c:229 (flb-rt-filter_grep+0x8ecb9)
    #1 flb_test_filter_grep_invalid /home/taka/git/fluent-bit/tests/runtime/filter_grep.c:144 (flb-rt-filter_grep+0x8e196)
    #2 test_do_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1007 (flb-rt-filter_grep+0x8b315)
    #3 test_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1103 (flb-rt-filter_grep+0x8b699)
    #4 main /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1700 (flb-rt-filter_grep+0x8d0f3)

  Previous write of size 4 at 0x7ba400000038 by thread T1:
    #0 flb_engine_shutdown /home/taka/git/fluent-bit/src/flb_engine.c:736 (flb-rt-filter_grep+0xb0026)
    #1 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:631 (flb-rt-filter_grep+0x90394)

  Location is heap block of size 18184 at 0x7ba400000000 allocated by main thread:
    #0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:668 (libtsan.so.0+0x305aa)
    #1 flb_calloc /home/taka/git/fluent-bit/include/fluent-bit/flb_mem.h:78 (flb-rt-filter_grep+0x9f314)
    #2 flb_config_init /home/taka/git/fluent-bit/src/flb_config.c:150 (flb-rt-filter_grep+0xa1f27)
    #3 flb_create /home/taka/git/fluent-bit/src/flb_lib.c:152 (flb-rt-filter_grep+0x8e8d7)
    #4 flb_test_filter_grep_invalid /home/taka/git/fluent-bit/tests/runtime/filter_grep.c:114 (flb-rt-filter_grep+0x8de0e)
    #5 test_do_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1007 (flb-rt-filter_grep+0x8b315)
    #6 test_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1103 (flb-rt-filter_grep+0x8b699)
    #7 main /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1700 (flb-rt-filter_grep+0x8d0f3)

  Thread T1 'flb-pipeline' (tid=21270, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (flb-rt-filter_grep+0x68429a)
    #2 flb_test_filter_grep_invalid /home/taka/git/fluent-bit/tests/runtime/filter_grep.c:133 (flb-rt-filter_grep+0x8e084)
    #3 test_do_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1007 (flb-rt-filter_grep+0x8b315)
    #4 test_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1103 (flb-rt-filter_grep+0x8b699)
    #5 main /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1700 (flb-rt-filter_grep+0x8d0f3)

SUMMARY: ThreadSanitizer: data race /home/taka/git/fluent-bit/src/flb_lib.c:229 in flb_destroy
==================
==================
WARNING: ThreadSanitizer: data race (pid=21267)
  Write of size 8 at 0x7b2000000e48 by main thread:
    #0 free ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:707 (libtsan.so.0+0x35f25)
    #1 flb_free /home/taka/git/fluent-bit/include/fluent-bit/flb_mem.h:122 (flb-rt-filter_grep+0x9f366)
    #2 flb_plugins_unregister /home/taka/git/fluent-bit/include/fluent-bit/flb_plugins.h:778 (flb-rt-filter_grep+0xa1d79)
    #3 flb_config_exit /home/taka/git/fluent-bit/src/flb_config.c:408 (flb-rt-filter_grep+0xa2e28)
    #4 flb_destroy /home/taka/git/fluent-bit/src/flb_lib.c:232 (flb-rt-filter_grep+0x8ed01)
    #5 flb_test_filter_grep_invalid /home/taka/git/fluent-bit/tests/runtime/filter_grep.c:144 (flb-rt-filter_grep+0x8e196)
    #6 test_do_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1007 (flb-rt-filter_grep+0x8b315)
    #7 test_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1103 (flb-rt-filter_grep+0x8b699)
    #8 main /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1700 (flb-rt-filter_grep+0x8d0f3)

  Previous read of size 8 at 0x7b2000000e48 by thread T1:
    #0 flb_input_pause_all /home/taka/git/fluent-bit/src/flb_input.c:876 (flb-rt-filter_grep+0x955e6)
    #1 flb_engine_shutdown /home/taka/git/fluent-bit/src/flb_engine.c:737 (flb-rt-filter_grep+0xb003d)
    #2 flb_lib_worker /home/taka/git/fluent-bit/src/flb_lib.c:631 (flb-rt-filter_grep+0x90394)

  Thread T1 'flb-pipeline' (tid=21270, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
    #1 mk_utils_worker_spawn /home/taka/git/fluent-bit/lib/monkey/mk_core/mk_utils.c:284 (flb-rt-filter_grep+0x68429a)
    #2 flb_test_filter_grep_invalid /home/taka/git/fluent-bit/tests/runtime/filter_grep.c:133 (flb-rt-filter_grep+0x8e084)
    #3 test_do_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1007 (flb-rt-filter_grep+0x8b315)
    #4 test_run_ /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1103 (flb-rt-filter_grep+0x8b699)
    #5 main /home/taka/git/fluent-bit/tests/runtime/../lib/acutest/acutest.h:1700 (flb-rt-filter_grep+0x8d0f3)

SUMMARY: ThreadSanitizer: data race /home/taka/git/fluent-bit/include/fluent-bit/flb_mem.h:122 in flb_free
==================

[rittneje]

Thanks @nokute78! It seems there are actually two bugs in play.

1. flb_lib_worker should not set ctx->status at all, since that is flb_start's job. So we need to remove this line. Or at least, it should not be called in the failure case.
   

   fluent-bit/src/flb_lib.c

   Line 633 in 5d51473

   ctx->status = FLB_LIB_NONE;

2. The second race you found seems to be the same underlying issue that Valgrind found. Namely, flb_stop needs to be fixed to still call pthread_join even when ctx->status == FLB_LIB_ERROR. Alternatively, flb_start needs to arrange to call it whenever it is going to return failure.
3. The third race you found is another effect of (2). Basically, flb_destroy is freeing memory that flb_lib_worker uses, and it didn't wait for that thread to finish first.

My suspicion is that the third race is directly responsible for these test failures.

I think the simplest approach is to make flb_start call pthread_join. That alone would resolve all three races without any further changes. It would just need to call it in the two error returns before setting ctx->status.

fluent-bit/src/flb_lib.c

Lines 670 to 673 in 5d51473

	if (bytes <= 0) {
	ctx->status = FLB_LIB_ERROR;
	return -1;
	}

fluent-bit/src/flb_lib.c

Lines 680 to 684 in 5d51473

	else if (val == FLB_ENGINE_FAILED) {
	flb_error("[lib] backend failed");
	ctx->status = FLB_LIB_ERROR;
	return -1;
	}