Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: potential missing on_closed message on client-side #2525

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

kaijchen
Copy link

@kaijchen kaijchen commented Jan 28, 2024

What problem does this PR solve?

Issue Number:

Problem Summary:

We have met a problem in Apache Doris where a stream rpc client cannot receive any message (including on_closed) from stream server.

apache/doris#30476

In certain cases, if the server-side stream is actively closing the stream.
The close frame is not sent to client-side due to connected is not set.
We should set connected before send stream data.

What is changed and the side effects?

Changed:

Side effects:

  • Performance effects(性能影响):

  • Breaking backward compatibility(向后兼容性):


Check List:

  • Please make sure your changes are compilable(请确保你的更改可以通过编译).
  • When providing us with a new feature, it is best to add related tests(如果你向我们增加一个新的功能, 请添加相关测试).
  • Please follow Contributor Covenant Code of Conduct.(请遵循贡献者准则).

@chenBright
Copy link
Contributor

之前UT有问题。最新master分支已修复,可以更新试试。

@@ -247,6 +247,11 @@ void SendRpcResponse(int64_t correlation_id,
// Send rpc response over stream even if server side failed to create
// stream for some reason.
if(cntl->has_remote_stream()){
if (stream_ptr) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

能不能加一个注释说明下为什么SetConnected一定要在发消息之前设置吗?
如果这里有race的话,后面需要加一个barrier不?

Copy link
Author

@kaijchen kaijchen Feb 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

加了,我们在单机测试下,出现了客户端收不到 on_closed 的问题:

  1. 服务端 send response 先发了包,但还未设置 connected。
  2. 客户端收到 open 成功,处理完后续逻辑,执行 stream close。
  3. 服务端收到 on_closed 时候,set connected 还未执行,导致 close frame 没发出去。

img_v3_027h_ef451b0b-127e-421e-aa49-7a069148d34g

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

代码和注释更新了,请再看一下。

Co-authored-by: Xin Liao <liaoxinbit@126.com>
@kaijchen kaijchen changed the title Fix: set connected before send stream data Fix: potential missing on_closed message on client-side Mar 25, 2024
@wwbmmm
Copy link
Contributor

wwbmmm commented Apr 1, 2024

StreamingRpcTest.server_send_data_before_run_done 这个单测失败了,是不是和改动有关?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants