Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

brpc申请8k导致tcmalloc出core #1979

Closed
vinllen opened this issue Nov 1, 2022 · 3 comments
Closed

brpc申请8k导致tcmalloc出core #1979

vinllen opened this issue Nov 1, 2022 · 3 comments

Comments

@vinllen
Copy link

vinllen commented Nov 1, 2022

Describe the bug (描述bug)
core栈:

(gdb) bt
#0  SLL_Next (t=0xe000000038000000) at infra-cpp-thirdparty/gperftools-27/src/linked_list.h:69
#1  SLL_TryPop (rv=<synthetic pointer>, list=0x31b1dc0) at infra-cpp-thirdparty/gperftools-27/src/linked_list.h:69
#2  TryPop (rv=<synthetic pointer>, this=0x31b1dc0) at infra-cpp-thirdparty/gperftools-27/src/thread_cache.h:220
#3  Allocate (oom_handler=0x152fac0 <tcmalloc::malloc_oom(unsigned long)>, cl=48, size=8192, this=0x31b17c0) at infra-cpp-thirdparty/gperftools-27/src/thread_cache.h:379
#4  malloc_fast_path<tcmalloc::malloc_oom> (size=8192) at infra-cpp-thirdparty/gperftools-27/src/tcmalloc.cc:1855
#5  tc_malloc (size=8192) at infra-cpp-thirdparty/gperftools-27/src/tcmalloc.cc:1880
#6  0x0000000000e9957e in create_block (block_size=8192) at infra-cpp-thirdparty/brpc-096/src/butil/iobuf.cpp:459
#7  create_block (block_size=8192) at infra-cpp-thirdparty/brpc-096/src/butil/iobuf.cpp:297
#8  create_block () at infra-cpp-thirdparty/brpc-096/src/butil/iobuf.cpp:311
#9  butil::iobuf::acquire_tls_block () at infra-cpp-thirdparty/brpc-096/src/butil/iobuf.cpp:459
#10 0x0000000000e9d037 in butil::IOPortal::pappend_from_file_descriptor(int, long, unsigned long) () at infra-cpp-thirdparty/brpc-096/src/butil/iobuf.cpp:1558
#11 0x0000000000d9841d in append_from_file_descriptor (max_count=524288, fd=<optimized out>, this=0x2cd66080) at infra-cpp-thirdparty/brpc-096/src/butil/iobuf_inl.h:42
#12 brpc::Socket::DoRead(unsigned long) () at infra-cpp-thirdparty/brpc-096/src/brpc/socket.cpp:1838
#13 0x0000000000ce63b3 in brpc::InputMessenger::OnNewMessages(brpc::Socket*) () at infra-cpp-thirdparty/brpc-096/src/brpc/input_messenger.cpp:202
#14 0x0000000000d9a9fd in brpc::Socket::ProcessEvent(void*) () at infra-cpp-thirdparty/brpc-096/src/brpc/socket.cpp:1020
#15 0x0000000000e230cf in bthread::TaskGroup::task_runner(long) () at infra-cpp-thirdparty/brpc-096/src/bthread/task_group.cpp:297
#16 0x0000000000e0caa1 in bthread_make_fcontext () at external/ks_build_tools/gcc-8.3.0/bin/../lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../include/c++/8.3.0/new:169
Cannot access memory at address 0x7f91bf428000
(gdb) f 1
#1  SLL_TryPop (rv=<synthetic pointer>, list=0x31b1dc0) at infra-cpp-thirdparty/gperftools-27/src/linked_list.h:69
69	in infra-cpp-thirdparty/gperftools-27/src/linked_list.h
(gdb) p list
$1 = (void **) 0x31b1dc0
(gdb) f 1
#1  SLL_TryPop (rv=<synthetic pointer>, list=0x31b1dc0) at infra-cpp-thirdparty/gperftools-27/src/linked_list.h:69
69	in infra-cpp-thirdparty/gperftools-27/src/linked_list.h
(gdb) p next
$2 = <optimized out>
(gdb) p list
$3 = (void **) 0x31b1dc0
(gdb) p *((void **) 0x31b1dc0)
$4 = (void *) 0xe000000038000000
(gdb)

To Reproduce (复现方法)
偶现问题

Expected behavior (期望行为)

Versions (各种版本)
OS: 4.18
Compiler:
brpc:
protobuf:

Additional context/screenshots (更多上下文/截图)

gperftools/gperftools#1036 tcmalloc这里有相同反馈的问题,作者认为这是上层管理的bug。

@lorinlee
Copy link
Contributor

lorinlee commented Nov 9, 2022

应该是其他地方内存写坏导致破坏了tcmalloc的内部数据结构,挂在这里只是brpc申请内存触发了,这里已经不是第一现场了。可以用ASAN或者valgrind看看是不是哪里内存写坏了

@vinllen
Copy link
Author

vinllen commented Nov 10, 2022

了解 预期猜测的确是上层的问题,我再看下

@vinllen vinllen closed this as completed Nov 10, 2022
@tangzhiqiang3
Copy link

tangzhiqiang3 commented Sep 15, 2023

demo program:
`#include <stdio.h>
#include <gperftools/tcmalloc.h>

int main()
{
char *ptr = (char *)tc_malloc(sizeof(char));
if(ptr){
printf("tcmalloc works failed!\n");
}
tc_free(ptr);

printf("tcmalloc works fine.\n");

return 0;

}`

Execution issues after cross compilation:
ldd test
./test:
libtcmalloc.so.9 => /var/data/demo_test/libtcmalloc.so.9 (0x268c1ce000)
libc++.so.1 => /lib64/libc++.so.1 (0x268c3d1000)
libm.so.3 => /lib64/libm.so.3 (0x268c4c0000)
libc.so.4 => /usr/lib/ldqnx-64.so.2 (0x268c11b000)
./test

Process 905670781 (test) terminated SIGSEGV code=1 fltno=11 ip=0000000defa9a868(/var/data/demo_test/libtcmalloc.so.9@calloc+0x0000000000000000) mapaddr=000000000002e868. ref=000000368d29cfc0
Memory fault (core dumped)

debug:
(gdb) set solib-search-path ./lib
(gdb) file test
Reading symbols from test...done.
(gdb) core-file test.core
[New pid 39800955 tid 1]
Program terminated with signal SIGSEGV, Segmentation fault.
#0 tc_calloc (n=n@entry=34, elem_size=elem_size@entry=8) at src/tcmalloc.cc:1984
1984 size_t elem_size) PERFTOOLS_NOTHROW {
(gdb) f 1
#1 0x00000033358c4e04 in __emutls_get_address (obj=0x33358e4040 <__emutls_v._ZN8tcmalloc11ThreadCache17threadlocal_data_E>) at ../../../../libgcc/emutls.c:159
159 ../../../../libgcc/emutls.c: No such file or directory.
(gdb) f 1
#1 0x00000033358c4e04 in __emutls_get_address (obj=0x33358e4040 <__emutls_v._ZN8tcmalloc11ThreadCache17threadlocal_data_E>) at ../../../../libgcc/emutls.c:159
159 in ../../../../libgcc/emutls.c
(gdb) p list
No symbol "list" in current context.
(gdb) p next
$1 = 1
(gdb) p list
No symbol "list" in current context.
(gdb)

why ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants