Improve using of ASan and TSan in CMake build #599

PragmaTwice · 2022-05-20T05:40:06Z

Close #590.

PragmaTwice · 2022-05-20T05:52:01Z

./unittest aborted while executed with AddressSanitizer:

SUMMARY: AddressSanitizer: 11268648 byte(s) leaked in 67853 allocation(s).

ShooterIT · 2022-05-20T05:59:01Z

Thanks @PragmaTwice Cool, I want to this commit too long time.

./unittest aborted while executed with AddressSanitizer:
SUMMARY: AddressSanitizer: 11268648 byte(s) leaked in 67853 allocation(s).

Did TCL tests report errors?

PragmaTwice · 2022-05-20T06:03:39Z

Thanks @PragmaTwice Cool, I want to this commit too long time.
./unittest aborted while executed with AddressSanitizer:
SUMMARY: AddressSanitizer: 11268648 byte(s) leaked in 67853 allocation(s).
Did TCL tests report errors?

Seems like the server aborted by asan while running tcl tests, I will check its log.

*** [err]: Can't start the Redis server

PragmaTwice · 2022-05-20T06:16:42Z

~/incubator-kvrocks/tests/tcl/tests/tmp$ rg AddressSanitizer .
./server.605939.3/stderr
137:SUMMARY: AddressSanitizer: 684 byte(s) leaked in 8 allocation(s).

Seems much less than unit tests?

Full stderr:

$ cat server*/stderr
E0520 14:00:27.832743 609966 replication.cc:131] Write error while sending batch to slave: Broken pipe. batches: 0x2433380D0A02000000000000000100000003013105010D0B5F5F6E616D65737061636561060100000000620D0A
E0520 14:00:24.070605 609173 replication.cc:165] [replication] connection error/eof, reconnect the master
E0520 14:00:48.974653 616014 replication.cc:165] [replication] connection error/eof, reconnect the master
E0520 14:00:49.974891 616014 replication.cc:249] [replication] Failed to connect the master, err: Connection refused
E0520 14:00:48.974625 616015 replication.cc:165] [replication] connection error/eof, reconnect the master
E0520 14:00:58.170694 616749 replication.cc:689] [replication] Failed to parallel fetch files while fetch file err: read size: File exists
E0520 14:01:08.171394 616749 replication.cc:165] [replication] connection error/eof, reconnect the master
E0520 14:01:15.107672 619319 replication.cc:165] [replication] connection error/eof, reconnect the master
E0520 14:01:16.910632 619653 replication.cc:165] [replication] connection error/eof, reconnect the master
E0520 14:00:32.504879 610212 replication.cc:165] [replication] connection error/eof, reconnect the master

=================================================================
==608988==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 288 byte(s) in 3 object(s) allocated from:
    #0 0x7f43526aa811 in operator new(unsigned long) /usr/src/debug/gcc/libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x5620e100092f in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::allocate(unsigned long, void const*) /usr/include/c++/11.2.0/ext/new_allocator.h:127
    #2 0x5620e100084a in std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&, unsigned long) /usr/include/c++/11.2.0/bits/alloc_traits.h:460
    #3 0x5620e0ffff7f in std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate(unsigned long) /usr/include/c++/11.2.0/bits/stl_vector.h:346
    #4 0x5620e100ba24 in void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x128a24)
    #5 0x5620e100a450 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >& std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::emplace_back<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) /usr/include/c++/11.2.0/bits/vector.tcc:121
    #6 0x5620e12a898e in Lua::redisGenericCommand(lua_State*, int) /home/twice/incubator-kvrocks/src/scripting.cc:358
    #7 0x5620e12a84ba in Lua::redisCallCommand(lua_State*) /home/twice/incubator-kvrocks/src/scripting.cc:333
    #8 0x5620e1a0020e in luaD_precall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d20e)
    #9 0x5620e1a0e6bc in luaV_execute (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb2b6bc)
    #10 0x5620e1a004b4 in luaD_call (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d4b4)
    #11 0x5620e19fca65 in f_call (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb19a65)
    #12 0x5620e19ff4d7 in luaD_rawrunprotected (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1c4d7)
    #13 0x5620e1a008b4 in luaD_pcall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d8b4)
    #14 0x5620e19fcb1f in lua_pcall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb19b1f)
    #15 0x5620e12a7956 in Lua::evalGenericCommand(Redis::Connection*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /home/twice/incubator-kvrocks/src/scripting.cc:303
    #16 0x5620e115b69a in Redis::CommandEval::Execute(Server*, Redis::Connection*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x27869a)
    #17 0x5620e11d37fa in Redis::Connection::ExecuteCommands(std::vector<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > const&) /home/twice/incubator-kvrocks/src/redis_connection.cc:422
    #18 0x5620e11cda04 in Redis::Connection::OnRead(bufferevent*, void*) /home/twice/incubator-kvrocks/src/redis_connection.cc:89
    #19 0x5620e19dca64 in bufferevent_run_deferred_callbacks_unlocked /home/twice/incubator-kvrocks/build/_deps/libevent-src/bufferevent.c:208
    #20 0x5620e19e77c9 in event_process_active_single_queue /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1703
    #21 0x5620e19e7c7f in event_process_active /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1789
    #22 0x5620e19e83d2 in event_base_loop /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:2012
    #23 0x5620e19e7d1b in event_base_dispatch /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1823
    #24 0x5620e13679c4 in Worker::Run(std::thread::id) /home/twice/incubator-kvrocks/src/worker.cc:234
    #25 0x5620e136c78e in operator() /home/twice/incubator-kvrocks/src/worker.cc:435
    #26 0x5620e136d887 in __invoke_impl<void, WorkerThread::Start()::<lambda()> > /usr/include/c++/11.2.0/bits/invoke.h:61
    #27 0x5620e136d84a in __invoke<WorkerThread::Start()::<lambda()> > /usr/include/c++/11.2.0/bits/invoke.h:96
    #28 0x5620e136d7f7 in _M_invoke<0> /usr/include/c++/11.2.0/bits/std_thread.h:253
    #29 0x5620e136d7cb in operator() /usr/include/c++/11.2.0/bits/std_thread.h:260

Direct leak of 84 byte(s) in 1 object(s) allocated from:
    #0 0x7f43526aa811 in operator new(unsigned long) /usr/src/debug/gcc/libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x5620e1aa8b4e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xbc5b4e)

Direct leak of 48 byte(s) in 1 object(s) allocated from:
    #0 0x7f43526aa811 in operator new(unsigned long) /usr/src/debug/gcc/libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x5620e116239c in Redis::redisCommandTable::{lambda()#75}::operator()() const (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x27f39c)
    #2 0x5620e11bb3f5 in std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> > std::__invoke_impl<std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> >, Redis::redisCommandTable::{lambda()#75}&>(std::__invoke_other, Redis::redisCommandTable::{lambda()#75}&) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x2d83f5)
    #3 0x5620e11a8ea7 in std::enable_if<is_invocable_r_v<std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> >, Redis::redisCommandTable::{lambda()#75}&>, std::enable_if>::type std::__invoke_r<std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> >, Redis::redisCommandTable::{lambda()#75}&>(std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> >&&, (Redis::redisCommandTable::{lambda()#75}&)...) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x2c5ea7)
    #4 0x5620e1191611 in std::_Function_handler<std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> > (), Redis::redisCommandTable::{lambda()#75}>::_M_invoke(std::_Any_data const&) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x2ae611)
    #5 0x5620e12aeff9 in std::function<std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> > ()>::operator()() const /usr/include/c++/11.2.0/bits/std_function.h:560
    #6 0x5620e12a8d67 in Lua::redisGenericCommand(lua_State*, int) /home/twice/incubator-kvrocks/src/scripting.cc:373
    #7 0x5620e12a84ba in Lua::redisCallCommand(lua_State*) /home/twice/incubator-kvrocks/src/scripting.cc:333
    #8 0x5620e1a0020e in luaD_precall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d20e)
    #9 0x5620e1a0e6bc in luaV_execute (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb2b6bc)
    #10 0x5620e1a004b4 in luaD_call (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d4b4)
    #11 0x5620e19fca65 in f_call (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb19a65)
    #12 0x5620e19ff4d7 in luaD_rawrunprotected (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1c4d7)
    #13 0x5620e1a008b4 in luaD_pcall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d8b4)
    #14 0x5620e19fcb1f in lua_pcall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb19b1f)
    #15 0x5620e12a7956 in Lua::evalGenericCommand(Redis::Connection*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /home/twice/incubator-kvrocks/src/scripting.cc:303
    #16 0x5620e115b69a in Redis::CommandEval::Execute(Server*, Redis::Connection*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x27869a)
    #17 0x5620e11d37fa in Redis::Connection::ExecuteCommands(std::vector<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > const&) /home/twice/incubator-kvrocks/src/redis_connection.cc:422
    #18 0x5620e11cda04 in Redis::Connection::OnRead(bufferevent*, void*) /home/twice/incubator-kvrocks/src/redis_connection.cc:89
    #19 0x5620e19dca64 in bufferevent_run_deferred_callbacks_unlocked /home/twice/incubator-kvrocks/build/_deps/libevent-src/bufferevent.c:208
    #20 0x5620e19e77c9 in event_process_active_single_queue /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1703
    #21 0x5620e19e7c7f in event_process_active /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1789
    #22 0x5620e19e83d2 in event_base_loop /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:2012
    #23 0x5620e19e7d1b in event_base_dispatch /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1823
    #24 0x5620e13679c4 in Worker::Run(std::thread::id) /home/twice/incubator-kvrocks/src/worker.cc:234
    #25 0x5620e136c78e in operator() /home/twice/incubator-kvrocks/src/worker.cc:435
    #26 0x5620e136d887 in __invoke_impl<void, WorkerThread::Start()::<lambda()> > /usr/include/c++/11.2.0/bits/invoke.h:61
    #27 0x5620e136d84a in __invoke<WorkerThread::Start()::<lambda()> > /usr/include/c++/11.2.0/bits/invoke.h:96
    #28 0x5620e136d7f7 in _M_invoke<0> /usr/include/c++/11.2.0/bits/std_thread.h:253
    #29 0x5620e136d7cb in operator() /usr/include/c++/11.2.0/bits/std_thread.h:260

Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x7f43526aa811 in operator new(unsigned long) /usr/src/debug/gcc/libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x5620e115f418 in Redis::redisCommandTable::{lambda()#35}::operator()() const (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x27c418)
    #2 0x5620e11b8ba5 in std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> > std::__invoke_impl<std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> >, Redis::redisCommandTable::{lambda()#35}&>(std::__invoke_other, Redis::redisCommandTable::{lambda()#35}&) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x2d5ba5)
    #3 0x5620e11a4357 in std::enable_if<is_invocable_r_v<std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> >, Redis::redisCommandTable::{lambda()#35}&>, std::enable_if>::type std::__invoke_r<std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> >, Redis::redisCommandTable::{lambda()#35}&>(std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> >&&, (Redis::redisCommandTable::{lambda()#35}&)...) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x2c1357)
    #4 0x5620e118d9e9 in std::_Function_handler<std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> > (), Redis::redisCommandTable::{lambda()#35}>::_M_invoke(std::_Any_data const&) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x2aa9e9)
    #5 0x5620e12aeff9 in std::function<std::unique_ptr<Redis::Commander, std::default_delete<Redis::Commander> > ()>::operator()() const /usr/include/c++/11.2.0/bits/std_function.h:560
    #6 0x5620e12a8d67 in Lua::redisGenericCommand(lua_State*, int) /home/twice/incubator-kvrocks/src/scripting.cc:373
    #7 0x5620e12a84ba in Lua::redisCallCommand(lua_State*) /home/twice/incubator-kvrocks/src/scripting.cc:333
    #8 0x5620e1a0020e in luaD_precall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d20e)
    #9 0x5620e1a0e6bc in luaV_execute (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb2b6bc)
    #10 0x5620e1a004b4 in luaD_call (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d4b4)
    #11 0x5620e19fca65 in f_call (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb19a65)
    #12 0x5620e19ff4d7 in luaD_rawrunprotected (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1c4d7)
    #13 0x5620e1a008b4 in luaD_pcall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d8b4)
    #14 0x5620e19fcb1f in lua_pcall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb19b1f)
    #15 0x5620e12a7956 in Lua::evalGenericCommand(Redis::Connection*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /home/twice/incubator-kvrocks/src/scripting.cc:303
    #16 0x5620e115b69a in Redis::CommandEval::Execute(Server*, Redis::Connection*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x27869a)
    #17 0x5620e11d37fa in Redis::Connection::ExecuteCommands(std::vector<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > const&) /home/twice/incubator-kvrocks/src/redis_connection.cc:422
    #18 0x5620e11cda04 in Redis::Connection::OnRead(bufferevent*, void*) /home/twice/incubator-kvrocks/src/redis_connection.cc:89
    #19 0x5620e19dca64 in bufferevent_run_deferred_callbacks_unlocked /home/twice/incubator-kvrocks/build/_deps/libevent-src/bufferevent.c:208
    #20 0x5620e19e77c9 in event_process_active_single_queue /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1703
    #21 0x5620e19e7c7f in event_process_active /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1789
    #22 0x5620e19e83d2 in event_base_loop /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:2012
    #23 0x5620e19e7d1b in event_base_dispatch /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1823
    #24 0x5620e13679c4 in Worker::Run(std::thread::id) /home/twice/incubator-kvrocks/src/worker.cc:234
    #25 0x5620e136c78e in operator() /home/twice/incubator-kvrocks/src/worker.cc:435
    #26 0x5620e136d887 in __invoke_impl<void, WorkerThread::Start()::<lambda()> > /usr/include/c++/11.2.0/bits/invoke.h:61
    #27 0x5620e136d84a in __invoke<WorkerThread::Start()::<lambda()> > /usr/include/c++/11.2.0/bits/invoke.h:96
    #28 0x5620e136d7f7 in _M_invoke<0> /usr/include/c++/11.2.0/bits/std_thread.h:253
    #29 0x5620e136d7cb in operator() /usr/include/c++/11.2.0/bits/std_thread.h:260

Indirect leak of 224 byte(s) in 2 object(s) allocated from:
    #0 0x7f43526aa811 in operator new(unsigned long) /usr/src/debug/gcc/libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x5620e100092f in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::allocate(unsigned long, void const*) /usr/include/c++/11.2.0/ext/new_allocator.h:127
    #2 0x5620e100084a in std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&, unsigned long) /usr/include/c++/11.2.0/bits/alloc_traits.h:460
    #3 0x5620e0ffff7f in std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate(unsigned long) /usr/include/c++/11.2.0/bits/stl_vector.h:346
    #4 0x5620e11d71f8 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >* std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >(unsigned long, __gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, __gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >) /usr/include/c++/11.2.0/bits/stl_vector.h:1511
    #5 0x5620e11d58a4 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::operator=(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /usr/include/c++/11.2.0/bits/vector.tcc:226
    #6 0x5620e11d5126 in Redis::Commander::SetArgs(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /home/twice/incubator-kvrocks/src/redis_cmd.h:63
    #7 0x5620e12a8daf in Lua::redisGenericCommand(lua_State*, int) /home/twice/incubator-kvrocks/src/scripting.cc:375
    #8 0x5620e12a84ba in Lua::redisCallCommand(lua_State*) /home/twice/incubator-kvrocks/src/scripting.cc:333
    #9 0x5620e1a0020e in luaD_precall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d20e)
    #10 0x5620e1a0e6bc in luaV_execute (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb2b6bc)
    #11 0x5620e1a004b4 in luaD_call (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d4b4)
    #12 0x5620e19fca65 in f_call (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb19a65)
    #13 0x5620e19ff4d7 in luaD_rawrunprotected (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1c4d7)
    #14 0x5620e1a008b4 in luaD_pcall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb1d8b4)
    #15 0x5620e19fcb1f in lua_pcall (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0xb19b1f)
    #16 0x5620e12a7956 in Lua::evalGenericCommand(Redis::Connection*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /home/twice/incubator-kvrocks/src/scripting.cc:303
    #17 0x5620e115b69a in Redis::CommandEval::Execute(Server*, Redis::Connection*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (/home/twice/incubator-kvrocks/tests/tcl/redis-server+0x27869a)
    #18 0x5620e11d37fa in Redis::Connection::ExecuteCommands(std::vector<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > const&) /home/twice/incubator-kvrocks/src/redis_connection.cc:422
    #19 0x5620e11cda04 in Redis::Connection::OnRead(bufferevent*, void*) /home/twice/incubator-kvrocks/src/redis_connection.cc:89
    #20 0x5620e19dca64 in bufferevent_run_deferred_callbacks_unlocked /home/twice/incubator-kvrocks/build/_deps/libevent-src/bufferevent.c:208
    #21 0x5620e19e77c9 in event_process_active_single_queue /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1703
    #22 0x5620e19e7c7f in event_process_active /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1789
    #23 0x5620e19e83d2 in event_base_loop /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:2012
    #24 0x5620e19e7d1b in event_base_dispatch /home/twice/incubator-kvrocks/build/_deps/libevent-src/event.c:1823
    #25 0x5620e13679c4 in Worker::Run(std::thread::id) /home/twice/incubator-kvrocks/src/worker.cc:234
    #26 0x5620e136c78e in operator() /home/twice/incubator-kvrocks/src/worker.cc:435
    #27 0x5620e136d887 in __invoke_impl<void, WorkerThread::Start()::<lambda()> > /usr/include/c++/11.2.0/bits/invoke.h:61
    #28 0x5620e136d84a in __invoke<WorkerThread::Start()::<lambda()> > /usr/include/c++/11.2.0/bits/invoke.h:96
    #29 0x5620e136d7f7 in _M_invoke<0> /usr/include/c++/11.2.0/bits/std_thread.h:253

SUMMARY: AddressSanitizer: 684 byte(s) leaked in 8 allocation(s).
E0520 14:00:23.513823 607579 redis_cmd.cc:1545] Failed to execute redis command: blpop, err: Invalid argument: WRONGTYPE Operation against a key holding the wrong kind of value
E0520 14:00:25.520509 607579 redis_cmd.cc:1545] Failed to execute redis command: brpop, err: Invalid argument: WRONGTYPE Operation against a key holding the wrong kind of value
E0520 14:01:02.859170 616953 slot_migrate.cc:474] [migrate] Failed to read response, Err: Connection reset by peer
E0520 14:01:02.859290 616953 slot_migrate.cc:793] [migrate] Wrong response
E0520 14:01:02.859315 616953 slot_migrate.cc:589] [migrate] Failed to migrate complex key: 0QG
E0520 14:01:02.859326 616953 slot_migrate.cc:305] [migrate] Failed to migrate key: 0QG
E0520 14:01:02.859344 616953 slot_migrate.cc:176] [migrate] Failed to send snapshot
E0520 14:01:02.859366 616953 slot_migrate.cc:416] [migrate] Failed to send import command to destination, slot: 8, error: Broken pipe
E0520 14:01:06.087435 617474 slot_migrate.cc:492] [migrate] Got invalid response: -MOVED 10 127.0.0.1:27121, line length: 25
E0520 14:01:06.087604 617474 slot_migrate.cc:793] [migrate] Wrong response
E0520 14:01:06.087630 617474 slot_migrate.cc:589] [migrate] Failed to migrate complex key: 7Ok
E0520 14:01:06.087641 617474 slot_migrate.cc:305] [migrate] Failed to migrate key: 7Ok
E0520 14:01:06.087657 617474 slot_migrate.cc:176] [migrate] Failed to send snapshot
E0520 14:01:08.262825 617856 slot_migrate.cc:283] [migrate] Stop migrating snapshot due to the thread stoped
E0520 14:01:08.262943 617856 slot_migrate.cc:176] [migrate] Failed to send snapshot
E0520 14:01:11.335420 618280 slot_migrate.cc:492] [migrate] Got invalid response: -NOAUTH Authentication required., line length: 32
E0520 14:01:11.335573 618280 slot_migrate.cc:251] [migrate] Failed to notify the destination to prepare to import data
E0520 14:01:11.335582 618280 slot_migrate.cc:165] [migrate] Failed to start migrating
E0520 14:01:11.335717 618280 slot_migrate.cc:492] [migrate] Got invalid response: -NOAUTH Authentication required., line length: 32
E0520 14:01:12.337333 618280 slot_migrate.cc:492] [migrate] Got invalid response: -ERR invalid password, line length: 21
E0520 14:01:12.337352 618280 slot_migrate.cc:402] [migrate] Failed to auth destination server with 'pass', stop migrating slot 22
E0520 14:01:12.337359 618280 slot_migrate.cc:165] [migrate] Failed to start migrating
E0520 14:01:12.337421 618280 slot_migrate.cc:492] [migrate] Got invalid response: -NOAUTH Authentication required., line length: 32
E0520 14:00:25.863554 610222 slot_migrate.cc:236] [migrate] Failed to connect destination server
E0520 14:00:25.863750 610222 slot_migrate.cc:165] [migrate] Failed to start migrating

PragmaTwice · 2022-05-20T06:21:43Z

But a weird thing is that the failed redis-server reported by tcl test has an empty stderr file.

git-hulk · 2022-05-20T13:26:54Z

Those memory leaks Looks all came from Lua scripting, I found we didn't close Lua VM before stopping the server. I will have a try and fix it soon.

PragmaTwice · 2022-05-20T13:36:18Z

seems like some of leaks from unittest is due to missing the dtor call (i.e. delete operator) after Storage::Open

git-hulk · 2022-05-22T13:55:24Z

@PragmaTwice How can I reproduce the memory leaks in tcl tests? I have tried to build with ENABLE_ASAN and run tcl tests, but can't find leaks error.

PragmaTwice · 2022-05-22T14:24:02Z

@PragmaTwice How can I reproduce the memory leaks in tcl tests? I have tried to build with ENABLE_ASAN and run tcl tests, but can't find leaks error.

Did these tcl tests on your side pass or fail?

There are some suggestions you could try:

make sure that your ./build/kvrocks is built with -DENABLE_ASAN=ON and from this branch
execute runtest with --dont-clean, so you can view stdout/stderr of kvrocks after test (grep AddressSanitizer . -r in ./tests/tcl)
I am using GCC 11.2 and Clang 13 in kernel 5.10, and they both can make the test fail, so you could try these version

stderr from Clang:

$ rg AddressSanitizer . 
./tests/tmp.bak/server.605939.3/stderr
137:SUMMARY: AddressSanitizer: 684 byte(s) leaked in 8 allocation(s).

./tests/tmp/server.724500.3/stderr
2:==726298==ERROR: AddressSanitizer: heap-use-after-free on address 0x608000020068 at pc 0x56231866985d bp 0x7fc7b4180570 sp 0x7fc7b4180568
81:SUMMARY: AddressSanitizer: heap-use-after-free /home/twice/incubator-kvrocks/src/redis_cmd.cc:1531:25 in Redis::CommandBPop::TryPopFromList()

./tests/tmp/server.724501.7/stderr
137:SUMMARY: AddressSanitizer: 684 byte(s) leaked in 8 allocation(s).

git-hulk · 2022-05-22T14:31:26Z

Copy that, thanks @PragmaTwice.

tisonkun · 2022-05-26T14:51:23Z

@PragmaTwice since #605 merged, is this PR ready for review (merge)?

PragmaTwice · 2022-05-26T15:21:41Z

@PragmaTwice since #605 merged, is this PR ready for review (merge)?

Leaks in unit tests are fixed, but these are still several leaks in TCL tests.
I will investigate it.

tisonkun · 2022-06-01T14:32:57Z

#614 is resolved now 🎉

tisonkun · 2022-06-02T03:22:15Z

@PragmaTwice shall we merge master, running CI with nightly changes first, or current status is already ready for review?

PragmaTwice · 2022-06-02T04:42:11Z

@PragmaTwice shall we merge master, running CI with nightly changes first, or current status is already ready for review?

Yeah, I think this PR is ready for review now~

git-hulk

LGTM, I love this PR.

ShooterIT

Truly thank you @PragmaTwice I waited too long time for this commit.

But i still want to make sure these sanitizers work well, can they report errors if we leak memory or data race access designedly

tisonkun · 2022-06-02T06:31:06Z

@ShooterIT so far, sanitizers run daily and there will be failure reports in "Actions" page if daily CI failed.

PragmaTwice · 2022-06-02T07:11:27Z

Truly thank you @PragmaTwice I waited too long time for this commit.

But i still want to make sure these sanitizers work well, can they report errors if we leak memory or data race access designedly

Yeah, I think these sanitizers work well.
Actually we have fixed two memory leaks found by the LeakSanitizer (#605 and #614).
Maybe later we can add more matrix strategies to CI so that both gcc and clang sanitizers can be used in tests.

PragmaTwice · 2022-06-02T07:14:43Z

One problem though is that the TCL tests seem not to pay attention to the return value of the redis server (kvrocks) when it crashes, which may make the sanitizer's error difficult to observe by developers, I will investigate this issue later.

tisonkun · 2022-06-02T08:20:09Z

One problem though is that the TCL tests seem not to pay attention to the return value of the redis server (kvrocks) when it crashes, which may make the sanitizer's error difficult to observe by developers, I will investigate this issue later.

This seems a separated issue. Merging this PR ...

PragmaTwice added 2 commits May 20, 2022 13:30

Improve using of ASan and TSan in CMake build

65a095a

CI: fix make targets

85ce81c

tisonkun mentioned this pull request May 24, 2022

unittests: fix leaks reported by ASan #605

Merged

PragmaTwice mentioned this pull request May 27, 2022

Memory leaks in Lua::redisGenericCommand #609

Closed

2 tasks

This comment was marked as outdated.

Sign in to view

Merge branch 'unstable' into sanitizer

ac33402

tisonkun requested review from git-hulk, ShooterIT and tisonkun June 2, 2022 05:59

git-hulk approved these changes Jun 2, 2022

View reviewed changes

tisonkun approved these changes Jun 2, 2022

View reviewed changes

ShooterIT approved these changes Jun 2, 2022

View reviewed changes

tisonkun merged commit a13465b into apache:unstable Jun 2, 2022

git-hulk mentioned this pull request Jun 2, 2022

Fix didn't build the unittest target in the daily CI #617

Merged

tisonkun mentioned this pull request Jun 12, 2022

[NEW] Add sanitizer support #405

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve using of ASan and TSan in CMake build #599

Improve using of ASan and TSan in CMake build #599

PragmaTwice commented May 20, 2022

PragmaTwice commented May 20, 2022

ShooterIT commented May 20, 2022

PragmaTwice commented May 20, 2022 •

edited

Loading

PragmaTwice commented May 20, 2022 •

edited

Loading

PragmaTwice commented May 20, 2022

git-hulk commented May 20, 2022

PragmaTwice commented May 20, 2022

git-hulk commented May 22, 2022 •

edited

Loading

PragmaTwice commented May 22, 2022 •

edited

Loading

git-hulk commented May 22, 2022

tisonkun commented May 26, 2022

PragmaTwice commented May 26, 2022

This comment was marked as outdated.

tisonkun commented Jun 1, 2022

tisonkun commented Jun 2, 2022

PragmaTwice commented Jun 2, 2022

git-hulk left a comment

ShooterIT left a comment

tisonkun commented Jun 2, 2022

PragmaTwice commented Jun 2, 2022

PragmaTwice commented Jun 2, 2022 •

edited

Loading

tisonkun commented Jun 2, 2022

Improve using of ASan and TSan in CMake build #599

Improve using of ASan and TSan in CMake build #599

Conversation

PragmaTwice commented May 20, 2022

PragmaTwice commented May 20, 2022

ShooterIT commented May 20, 2022

PragmaTwice commented May 20, 2022 • edited Loading

PragmaTwice commented May 20, 2022 • edited Loading

PragmaTwice commented May 20, 2022

git-hulk commented May 20, 2022

PragmaTwice commented May 20, 2022

git-hulk commented May 22, 2022 • edited Loading

PragmaTwice commented May 22, 2022 • edited Loading

git-hulk commented May 22, 2022

tisonkun commented May 26, 2022

PragmaTwice commented May 26, 2022

This comment was marked as outdated.

tisonkun commented Jun 1, 2022

tisonkun commented Jun 2, 2022

PragmaTwice commented Jun 2, 2022

git-hulk left a comment

Choose a reason for hiding this comment

ShooterIT left a comment

Choose a reason for hiding this comment

tisonkun commented Jun 2, 2022

PragmaTwice commented Jun 2, 2022

PragmaTwice commented Jun 2, 2022 • edited Loading

tisonkun commented Jun 2, 2022

PragmaTwice commented May 20, 2022 •

edited

Loading

PragmaTwice commented May 20, 2022 •

edited

Loading

git-hulk commented May 22, 2022 •

edited

Loading

PragmaTwice commented May 22, 2022 •

edited

Loading

PragmaTwice commented Jun 2, 2022 •

edited

Loading