Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rdma_performance bazel support #1984

Merged
merged 2 commits into from
Nov 9, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/cn/rdma.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,14 @@ mkdir bld && cd bld && cmake ..
make
```

使用bazel:
```bash
# Server
bazel build example:rdma_performance_server
# Client
bazel build example:rdma_performance_client
```

# 基本实现

RDMA与TCP不同,不使用socket接口进行通信。但是在实现上仍然复用了brpc中原本的Socket类。当用户选择ChannelOptions或ServerOptions中的use_rdma为true时,创建出的Socket类中则有对应的RdmaEndpoint(参见src/brpc/rdma/rdma_endpoint.cpp)。当RDMA被使能时,写入Socket的数据会通过RdmaEndpoint提交给RDMA QP(通过verbs API),而非拷贝到fd。对于数据读取,RdmaEndpoint中则调用verbs API从RDMA CQ中获取对应完成信息(事件获取有独立的fd,复用EventDispatcher,处理函数采用RdmaEndpoint::PollCq),最后复用InputMessenger完成RPC消息解析。
Expand Down
8 changes: 8 additions & 0 deletions docs/en/rdma.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,14 @@ mkdir bld && cd bld && cmake ..
make
```

With bazel:
```bash
# Server
bazel build example:rdma_performance_server
# Client
bazel build example:rdma_performance_client
```

# Basic Implementation

RDMA does not use socket API like TCP. However, the brpc::Socket class is still used. If a user sets ChannelOptions.use_rdma or ServerOptions.use_rdma to true, the Socket class created has RdmaEndpoint (see src/brpc/rdma/rdma_endpoint.cpp). When RDMA is enabled, the data which need to transmit will be posted to RDMA QP with verbs API, not written to TCP fd. For data receiving, RdmaEndpoint will get completions from RDMA CQ with verbs API (the event will be generated from a dedicated fd and be added into EventDispatcher, the handling function is RdmaEndpoint::PollCq) before parsing RPC messages with InputMessenger.
Expand Down
47 changes: 47 additions & 0 deletions example/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ COPTS = [
] + select({
"//bazel/config:brpc_with_glog": ["-DBRPC_WITH_GLOG=1"],
"//conditions:default": ["-DBRPC_WITH_GLOG=0"],
}) + select({
"//bazel/config:brpc_with_rdma": ["-DBRPC_WITH_RDMA=1"],
"//conditions:default": [""],
})

proto_library(
Expand All @@ -41,13 +44,27 @@ proto_library(
],
)

proto_library(
name = "rdma_performance_proto",
srcs = [
"rdma_performance/test.proto",
],
)

cc_proto_library(
name = "cc_echo_c++_proto",
deps = [
":echo_c++_proto",
],
)

cc_proto_library(
name = "cc_rdma_performance_proto",
deps = [
":rdma_performance_proto",
],
)

cc_binary(
name = "echo_c++_server",
srcs = [
Expand Down Expand Up @@ -77,3 +94,33 @@ cc_binary(
"//:brpc",
],
)

cc_binary(
name = "rdma_performance_server",
srcs = [
"rdma_performance/server.cpp",
],
includes = [
"rdma_performance",
],
copts = COPTS + ["-DBRPC_WITH_RDMA=1"],
deps = [
":cc_rdma_performance_proto",
"//:brpc",
],
)

cc_binary(
name = "rdma_performance_client",
srcs = [
"rdma_performance/client.cpp",
],
includes = [
"rdma_performance",
],
copts = COPTS + ["-DBRPC_WITH_RDMA=1"],
deps = [
":cc_rdma_performance_proto",
"//:brpc",
],
)