Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

使用asan检查程序内存时,brpc服务端函数内部的heap-use-after-free,无法提示内存被释放的完整堆栈 #2172

Closed
zzjshaok opened this issue Mar 21, 2023 · 3 comments

Comments

@zzjshaok
Copy link

zzjshaok commented Mar 21, 2023

Describe the bug (描述bug)
brpc服务端函数内部使用已经被释放的内存时,asan 无法显示内存释放的完整堆栈

To Reproduce (复现方法)
brpc 服务端函数如下

void HandlerServiceImpl::IndexReadMap(google::protobuf::RpcController* cntl_base,
                                      const IndexReadMapRequest* req, IndexReadMapResponse* resp,
                                      google::protobuf::Closure* done)
{
    brpc::ClosureGuard done_guard(done); 
    ...................
    SessionData *obj = new SessionData();
    obj->m_in_busy = 1;
    delete obj;
    obj->m_in_busy = 0;
    ...................
    return; 
}

程序开启asan, 运行后提示

ERROR: AddressSanitizer: heap-use-after-free on address 0x621000814dd0 at pc 0x000004668773 bp 0x7ff50aef2010 sp 0x7ff50aef2008
WRITE of size 4 at 0x621000814dd0 thread T108
    #0 0x4668772 in crab::HandlerServiceImpl::IndexReadMap(google::protobuf::RpcController*, crab::IndexReadMapRequest const*, crab::IndexReadMapResponse*, google::protobuf::Closure*) /data/test/crab_db/sql/crab/server/handler_service.cc:850
    #1 0x74a4285 in crab::HandlerService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*) /data/test/crab_db/sql/crab/proto/handler.pb.cc:33140
    #2 0x7619476 in brpc::policy::ProcessRpcRequest(brpc::InputMessageBase*) /data/test/crab_db/3rd/brpc/src/brpc/policy/baidu_rpc_protocol.cpp:513
    #3 0x760db36 in brpc::ProcessInputMessage(void*) /data/test/crab_db/3rd/brpc/src/brpc/input_messenger.cpp:159
    #4 0x760ef65 in brpc::InputMessenger::OnNewMessages(brpc::Socket*) /data/test/crab_db/3rd/brpc/src/brpc/input_messenger.cpp:334
    #5 0x76d1a3c in brpc::Socket::ProcessEvent(void*) /data/test/crab_db/3rd/brpc/src/brpc/socket.cpp:1093
    #6 0x75d78be in bthread::TaskGroup::task_runner(long) /data/test/crab_db/3rd/brpc/src/bthread/task_group.cpp:375
    #7 0x7758190 in bthread_make_fcontext (/data/test/ob_rel/crab_db/bin/crabdb+0x7758190)

0x621000814dd0 is located 208 bytes inside of 4376-byte region [0x621000814d00,0x621000815e18)
**freed by thread T108 here:
    #0 0x7ff56c744578 in operator delete(void*, unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cc:151**

previously allocated by thread T108 here:
    #0 0x7ff56c743180 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cc:90

Thread T108 created by T0 here:
    #0 0x7ff56c6a91b0 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cc:202
    #1 0x75c856f in bthread::TaskControl::add_workers(int) /data/test/crab_db/3rd/brpc/src/bthread/task_control.cpp:273
    #2 0x75c1d2b in bthread_setconcurrency /data/test/crab_db/3rd/brpc/src/bthread/bthread.cpp:316
    #3 0x76bfe5f in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) /data/test/crab_db/3rd/brpc/src/brpc/server.cpp:960
    #4 0x76c12df in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) /data/test/crab_db/3rd/brpc/src/brpc/server.cpp:1124
    #5 0x76c1467 in brpc::Server::Start(int, brpc::ServerOptions const*) /data/test/crab_db/3rd/brpc/src/brpc/server.cpp:1143
    #6 0x43ac7dd in crab::HandlerServer::Start() /data/test/crab_db/sql/crab/server/handler_server.cc:46
    #7 0x39d3065 in crab_plugin_init() /data/test/crab_db/sql/crab/plugin/init.cc:304
    #8 0x3312c87 in crabdb_main(int, char**) /data/test/crab_db/sql/crabdb.cc:8085
    #9 0x32ebdba in main /data/test/crab_db/sql/main.cc:25
	#10 0x7ff569b48554 in __libc_start_main (/lib64/libc.so.6+0x22554)

SUMMARY: AddressSanitizer: heap-use-after-free /data/test/crab_db/sql/crab/server/handler_service.cc:850 in crab::HandlerServiceImpl::IndexReadMap(google::protobuf::RpcController*, crab::IndexReadMapRequest const*, crab::IndexReadMapResponse*, google::protobuf::Closure*)
Shadow bytes around the buggy address:
  0x0c42800fa960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c42800fa970: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c42800fa980: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c42800fa990: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c42800fa9a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c42800fa9b0: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
  0x0c42800fa9c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c42800fa9d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c42800fa9e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c42800fa9f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c42800faa00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

Expected behavior (期望行为)
期望 freed by thread T108 here: 这里显示完整的堆栈,内存被程序的哪一行释放,方便定位问题。
而不仅仅是只有一行

 #0 0x7ff56c744578 in operator delete(void*, unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cc:151

如果是在brpc客户端发生同样的行为,asan可以提示出释放内存完整的调用堆栈。
brpc协程切换影响了asan的功能?

Versions (各种版本)
OS:
Compiler:
brpc:
protobuf:

Additional context/screenshots (更多上下文/截图)

@warriorpaw
Copy link
Contributor

warriorpaw commented Mar 27, 2023

增加--usercode_in_pthread=true命令行参数再执行,asan的堆栈应该就是完整的,see: google/sanitizers#189 ,asan不支持swapcontext,但是 bthread就是基于swapcontext 实现的,usercode_in_pthread参数会让brpc在pthread中执行。

@zzjshaok
Copy link
Author

增加--usercode_in_pthread=true命令行参数再执行,asan的堆栈应该就是完整的,see: google/sanitizers#189 ,asan不支持swapcontext,但是 bthread就是基于swapcontext 实现的,usercode_in_pthread参数会让brpc在pthread中执行。

谢谢!测试后确实是在这样的,加上--usercode_in_pthread=true后就可以显示完整堆栈了

@chenBright
Copy link
Contributor

chenBright commented Dec 31, 2024

经验证,设置fast_unwind_on_malloc=0就可以回溯到完整的调用栈了。需要注意的是fast_unwind_on_malloc=0很耗性能。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants