Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crash #985

Closed
zgb40302 opened this issue Dec 5, 2019 · 5 comments
Closed

crash #985

zgb40302 opened this issue Dec 5, 2019 · 5 comments

Comments

@zgb40302
Copy link

zgb40302 commented Dec 5, 2019

Describe the bug (描述bug)
#0 0x00007f8c26c211f7 in raise () from /lib64/libc.so.6
#1 0x00007f8c26c228e8 in abort () from /lib64/libc.so.6
#2 0x00000000011dc455 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x0000000001147756 in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:47
#4 0x00000000011525e9 in __cxa_call_terminate (ue_header=ue_header@entry=0x7f8b1b234410) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54
#5 0x0000000001146d88 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944,
ue_header=0x7f8b1b234410, context=) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:676
#6 0x00007f8c271da8a3 in ?? () from /lib64/libgcc_s.so.1
#7 0x00007f8c271dadd7 in _Unwind_Resume () from /lib64/libgcc_s.so.1
#8 0x0000000000685169 in operator() (this=, obj=) at /data1/incubator-brpc-0.9.6/src/brpc/destroyable.h:36
#9 ~unique_ptr (this=, __in_chrg=) at /usr/include/c++/4.8.2/bits/unique_ptr.h:184
#10 ~DestroyingPtr (this=, __in_chrg=) at /data1/incubator-brpc-0.9.6/src/brpc/destroyable.h:42
#11 brpc::policy::ProcessHttpRequest (msg=) at /data1/incubator-brpc-0.9.6/src/brpc/policy/http_rpc_protocol.cpp:1493
#12 0x0000000000650e4a in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7f8b1be72750) -- 指针是brpc::InputMessageBase类型
at /data1/incubator-brpc-0.9.6/src/brpc/input_messenger.cpp:136
#13 0x0000000000651c66 in operator() (this=, last_msg=0x7f8b1be72750)
at /data1/incubator-brpc-0.9.6/src/brpc/input_messenger.cpp:142
#14 brpc::InputMessenger::OnNewMessages (m=0x7f8ae401b080) at /usr/include/c++/4.8.2/bits/unique_ptr.h:184
#15 0x00000000005af71d in brpc::Socket::ProcessEvent (arg=0x7f8ae401b080) at /data1/incubator-brpc-0.9.6/src/brpc/socket.cpp:1020
#16 0x00000000005fe3bf in bthread::TaskGroup::task_runner (skip_remained=)
at /data1/incubator-brpc-0.9.6/src/bthread/task_group.cpp:297
#17 0x00000000005ecc71 in bthread_make_fcontext ()
#18 0x0000000000000000 in ?? ()

To Reproduce (复现方法)
random to hanppen

Expected behavior (期望行为)
fix

Versions (各种版本)
OS: centos 7.3
Compiler: gcc 4.8.5
brpc: 0.9.6
protobuf: 3.6.1.2

Additional context/screenshots (更多上下文/截图)
正常的流程应该是调用析构时不会再调用operator函数,而出问题的环境确实调用了operator,说明imsg_guard里面的内容没有变成NULL。正常的流程如下:
image

image

image
--1479行调用reset 之后,imsg_guard 里面的内容变成NULL了,

(gdb) n
std::unique_ptr<brpc::policy::HttpContext, brpc::detail::Destroyerbrpc::policy::HttpContext >::reset (this=0x7ffef84f5ac0, __p=0x7ffe70042520) at /usr/include/c++/4.8.2/bits/unique_ptr.h:263

image

(gdb) p *(brpc::DestroyingPtrbrpc::policy::HttpContext *) 0x7ffef84f5ac0
$52 = {
<std::unique_ptr<brpc::policy::HttpContext, brpc::detail::Destroyerbrpc::policy::HttpContext >> = std::unique_ptrbrpc::policy::HttpContext containing 0x0, }

为什么析构的时候还会调用operator()

std::unique_ptr<brpc::policy::HttpContext, brpc::detail::Destroyerbrpc::policy::HttpContext >::~unique_ptr (this=0x7ffeec4f5ac0, __in_chrg=) at /usr/include/c++/4.8.2/bits/unique_ptr.h:182

image

#8 0x0000000000685169 in operator() (this=, obj=) at /data1/incubator-brpc-0.9.6/src/brpc/destroyable.h:36
#9 ~unique_ptr (this=, __in_chrg=) at /usr/include/c++/4.8.2/bits/unique_ptr.h:184
#10 ~DestroyingPtr (this=, __in_chrg=) at /data1/incubator-brpc-0.9.6/src/brpc/destroyable.h:42
#11 brpc::policy::ProcessHttpRequest (msg=) at /data1/incubator-brpc-0.9.6/src/brpc/policy/http_rpc_protocol.cpp:1493

@jamesge
Copy link
Contributor

jamesge commented Dec 6, 2019

可以试一下: 1. 去掉-O2重新编译brpc然后尝试复现问题,可能有更准确的栈;2. 运行asan看看是否有内存方面的问题,此类问题大概率和应用代码的内存问题相关。

@zgb40302
Copy link
Author

zgb40302 commented Dec 13, 2019

1,asan 没有内存问题。
2,去掉 O2后,用的是O1,栈信息显示 operator() 的参数 obj=0x0,为啥还会core ?
35 template struct Destroyer {
36 void operator()(T* obj) const { if (obj) { obj->Destroy(); } }
37 };

#0 0x00007ff5608351f7 in raise () from /lib64/libc.so.6
#1 0x00007ff5608368e8 in abort () from /lib64/libc.so.6
#2 0x00000000011c12a5 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x000000000112c556 in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:47
#4 0x00000000011373e9 in __cxa_call_terminate (ue_header=ue_header@entry=0x7ff458517fd0) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54
#5 0x000000000112bb88 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=0x7ff458517fd0, context=)
at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:676
#6 0x00007ff560dee8a3 in ?? () from /lib64/libgcc_s.so.1
#7 0x00007ff560deedd7 in _Unwind_Resume () from /lib64/libgcc_s.so.1
#8 0x00000000006746c2 in operator() (obj=0x0, this=) at /data1/incubator-brpc/src/brpc/destroyable.h:36
#9 ~unique_ptr (this=, __in_chrg=) at /usr/include/c++/4.8.2/bits/unique_ptr.h:184
#10 ~DestroyingPtr (this=, __in_chrg=) at /data1/incubator-brpc/src/brpc/destroyable.h:42
#11 brpc::policy::ProcessHttpRequest (msg=) at /data1/incubator-brpc/src/brpc/policy/http_rpc_protocol.cpp:1493
#12 0x000000000064043a in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7ff45814cfc0) at /data1/incubator-brpc/src/brpc/input_messenger.cpp:136
#13 0x00000000006414b5 in operator() (last_msg=0x7ff45814cfc0, this=) at /data1/incubator-brpc/src/brpc/input_messenger.cpp:142
#14 brpc::InputMessenger::OnNewMessages (m=0x7ff4300f28e0) at /usr/include/c++/4.8.2/bits/unique_ptr.h:184
#15 0x00000000005a1249 in brpc::Socket::ProcessEvent (arg=0x7ff4300f28e0) at /data1/incubator-brpc/src/brpc/socket.cpp:1020
#16 0x00000000005ecd18 in bthread::TaskGroup::task_runner (skip_remained=) at /data1/incubator-brpc/src/bthread/task_group.cpp:297
#17 0x00000000005db941 in bthread_make_fcontext ()

@zgb40302
Copy link
Author

zgb40302 commented Dec 13, 2019

每次core的时候,/var/log/message 里都会记录以下日志:
messages-20191208:Dec 4 19:02:59 131-15-98-huawei-core sunflower: terminate called after throwing an instance of 'std::bad_alloc'
messages-20191208:Dec 4 19:02:59 131-15-98-huawei-core sunflower: what(): std::bad_alloc

@zgb40302
Copy link
Author

@jamesge 改成 O0 后, 问题已解决,确实是应用层问题。多谢指教。

@guodongxiaren
Copy link
Member

m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants