You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
int Socket::StartWrite(WriteRequest* req, const WriteOptions& opt, bool in_background) {
// Release fence makes sure the thread getting request sees *req
WriteRequest* const prev_head =
_write_head.exchange(req, butil::memory_order_release);
if (prev_head != NULL) {
// Someone is writing to the fd. The KeepWrite thread may spin
// until req->next to be non-UNCONNECTED. This process is not
// lock-free, but the duration is so short(1~2 instructions,
// depending on compiler) that the spin rarely occurs in practice
// (I've not seen any spin in highly contended tests).
req->next = prev_head;
return 0;
}
...
// Write once in the calling thread. If the write is not complete,
// continue it in KeepWrite thread.
if (_conn) {
butil::IOBuf* data_arr[1] = { &req->data };
nw = _conn->CutMessageIntoFileDescriptor(fd(), data_arr, 1);
} else {
nw = req->data.cut_into_file_descriptor(fd());
}
...
if (IsWriteComplete(req, true, NULL)) {
ReturnSuccessfulWriteRequest(req);
return 0;
}
KEEPWRITE_IN_BACKGROUND:
ReAddress(&ptr_for_keep_write);
req->socket = ptr_for_keep_write.release();
if (bthread_start_background(&th, &BTHREAD_ATTR_NORMAL,
KeepWrite, req) != 0) {
LOG(FATAL) << "Fail to start KeepWrite";
KeepWrite(req);
}
return 0;
}
我们在使用 brpc Stream 过程中观察到一个现象,单线程 StreamWrite 很慢,每秒只能发出几万条 message,而两个线程的话就可以达到每秒百万条。原因是只有一个线程的话每次 StreamWrite 都会同步写 socket。
StreamWrite 直接调用
Socket::StartWrite
,在 request 链为空时,插入并且直接写入 socket。同时并发写的其他人会把请求入队直接返回,负责写的线程在写完自己的数据后会再次检查是否有更多的数据,如果有则起一个背景 KEEPWRITE bthread 负责把所有的数据写入 socket。这个实现的问题在于单线程每次 StreamWrite 都是同步写。与普通 RPC 不同,Streaming RPC 的使用场景通常是单线程循环调用 StreamWrite 将一堆数据发出去。单线程 StreamWrite 每次都会直接往 socket 里写,而不会触发 KEEPWRITE 背景线程,我们这里观察到每次写耗时大约10-20us,导致单线程每秒最多只能发送出几万条 message。同步写导致 Stream 失去了 batch 的作用,性能退化为了普通 RPC。只有在多线程同时 StreamWrite 时才会触发背景 KEEPWRITE bthread,达到比较好的 batch 效果,每秒可以发送出几十到上百万条 message,两者性能天差地别。
我们的改动是对
StartWrite
增加了 write_in_background 参数,如果为 true,获得写权限的线程不再尝试直接写 socket 而是跳到KEEPWRITE_IN_BACKGROUND
启动背景线程来写,每次 StreamWrite 只是把数据插入进去后可以立即返回,立即往链表里插入新的数据,背景线程不断发现有新数据可写,每次 socket write 都可以写入一堆数据而不只是一条,可以很好地 batch,单线程性能就可以达到原本多线程的水平。另外
StartWrite
的直接写 socket 对普通的异步 RPC 也会有影响,每次 RPC 调用都会有 10-20us,在关键路径上也会是很大的开销,对于异步 RPC 也交给背景 bthread 写直接返回,感觉要好些。The text was updated successfully, but these errors were encountered: