Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

brpc::Channel::CallMethod阻塞,不超时 #1507

Closed
datatechboost opened this issue Aug 3, 2021 · 3 comments
Closed

brpc::Channel::CallMethod阻塞,不超时 #1507

datatechboost opened this issue Aug 3, 2021 · 3 comments

Comments

@datatechboost
Copy link

Describe the bug (描述bug)

模块a 3个rpc调用阻塞,这3个rpc都是对模块b的调用,共涉及模块b的两个方法。
channel初始化时option为null, rpc调用时 brpc::Controller上set_timeout_ms了。
3个rpc调用中的2个是在协程中进行的,另1个是在timer线程(pthread)调用的。

补充:brpc::channel的init是加了锁的。

To Reproduce (复现方法)

让模块b死锁,模块a发起3个模块b的rpc。

Expected behavior (期望行为)

3个rpc超时返回。

Versions (各种版本)
OS:CentOS Linux release 7.8.2003 (Core)
Compiler: gcc 版本 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
brpc:0.9.0
protobuf:3.0.0

Additional context/screenshots (更多上下文/截图)

@datatechboost
Copy link
Author

datatechboost commented Aug 3, 2021

3个堆栈的顶部,我已经使用内存检测工具排除了模块内存破坏的可能。

(gdb) bthread_frame 1
(gdb) bt
#0 0x00007f0157e4c95c in jump_stack (to=, from=) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/stack_inl.h:129
#1 bthread::TaskGroup::sched_to (pg=0x7f014802b810, pg@entry=0x7f010e55c470, next_meta=0x1, next_meta@entry=0x7f011800d420) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/task_group.cpp:603
#2 0x00007f0157e4e123 in sched_to (next_tid=, pg=0x7f010e55c470) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/task_group_inl.h:75
#3 bthread::TaskGroup::sched (pg=pg@entry=0x7f010e55c470) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/task_group.cpp:563
#4 0x00007f0157e3503b in bthread::butex_wait (arg=arg@entry=0x7f014802b590, expected_value=expected_value@entry=9, abstime=abstime@entry=0x0) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/butex.cpp:656
#5 0x00007f0157e3e790 in bthread_id_join (id=...) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/id.cpp:526
#6 0x00007f0157ebf45d in brpc::Channel::CallMethod (this=0x7f0118034418, method=0x7f011807e548, controller_base=0x7f010e55c740, request=0x7f010e55c6c0, response=0x7f010e55c700, done=0x0) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/brpc/channel.cpp:398

(gdb) bthread_frame 2
(gdb) bt
#0 0x00007f0157e4c95c in jump_stack (to=, from=) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/stack_inl.h:129
#1 bthread::TaskGroup::sched_to (pg=0x7f014802b810, pg@entry=0x7f010e056d20, next_meta=0x1, next_meta@entry=0x7f011000d420) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/task_group.cpp:603
#2 0x00007f0157e4e123 in sched_to (next_tid=, pg=0x7f010e056d20) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/task_group_inl.h:75
#3 bthread::TaskGroup::sched (pg=pg@entry=0x7f010e056d20) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/task_group.cpp:563
#4 0x00007f0157e3503b in bthread::butex_wait (arg=arg@entry=0x7f014802b410, expected_value=expected_value@entry=9, abstime=abstime@entry=0x0) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/butex.cpp:656
#5 0x00007f0157e3e790 in bthread_id_join (id=...) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/id.cpp:526
#6 0x00007f0157ebf45d in brpc::Channel::CallMethod (this=0x7f0118034418, method=0x7f011807e548, controller_base=0x7f010e056f90, request=0x7f010e056f10, response=0x7f010e056f50, done=0x0) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/brpc/channel.cpp:398

Thread 27 (Thread 0x7f0151e30700 (LWP 26847)):
#0 0x00007f0154751d19 in syscall () from /lib64/libc.so.6
#1 0x00007f0157e34283 in futex_wait_private (timeout=0x0, expected=0, addr1=0x7f0151e2f0e0) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/sys_futex.h:38
#2 bthread::wait_pthread (pw=..., ptimeout=ptimeout@entry=0x0) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/butex.cpp:138
#3 0x00007f0157e352bc in butex_wait_from_pthread (abstime=0x0, expected_value=1, b=0x7f014802b810, g=) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/butex.cpp:585
#4 bthread::butex_wait (arg=arg@entry=0x7f014802b810, expected_value=expected_value@entry=1, abstime=abstime@entry=0x0) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/butex.cpp:618
#5 0x00007f0157e3e790 in bthread_id_join (id=...) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/bthread/id.cpp:526
#6 0x00007f0157ebf45d in brpc::Channel::CallMethod (this=0x7f0118034418, method=0x7f011807e580, controller_base=0x7f0151e2f390, request=0x7f0151e2f2f0, response=0x7f0151e2f340, done=0x0) at /opt/data/code/vplatform_thirdparty/incubator-brpc-0.9.0/src/brpc/channel.cpp:398

(gdb) bthread_list 共6个
id tid function has stack total:6
#0 4294968066 0x7f0157f98b40 brpc::SocketMap::RunWatchConnections(void*) yes
#1 197568496390 0x7f0157f8cec0 brpc::Socket::ProcessEvent(void*) yes
#2 73014445571 0x7f0157f8cec0 brpc::Socket::ProcessEvent(void*) yes
#3 4294969600 0x7f0157ee92a0 brpc::GlobalUpdate(void*) yes
#4 4294969601 0x7f0157ee8d70 brpc::EventDispatcher::RunThis(void*) yes
#5 4294969602 0x7f0157f7f500 brpc::Server::UpdateDerivedVars(void*) yes

@chongg039
Copy link

@datatechboost hi请问这个问题后续有结论吗,我这边遇到了同样的问题

@wwbmmm
Copy link
Contributor

wwbmmm commented May 13, 2022

3个rpc调用中的2个是在协程中进行的,另1个是在timer线程(pthread)调用的。

timer 回调里是不能执行CallMethod,这样会把timer thread阻塞,然后超时也就失效了。

@wwbmmm wwbmmm closed this as completed Aug 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants