Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

支持pb3的arena分配 #535

Closed
jamesge opened this issue Oct 18, 2018 · 6 comments · Fixed by #2751
Closed

支持pb3的arena分配 #535

jamesge opened this issue Oct 18, 2018 · 6 comments · Fixed by #2751
Labels
enhancement improvements on existing features official created by brpc authors

Comments

@jamesge
Copy link
Contributor

jamesge commented Oct 18, 2018

百度内很难动态判定pb的版本并进行相应的操作,这点在开源版中可以放开一些,使用宏或其他编译时的手段来处理。arena对于大request的分配和使用应该有一定的性能提升。

@jamesge jamesge added enhancement improvements on existing features official created by brpc authors labels Oct 18, 2018
@gethinzhang
Copy link

目前看起来,是不是只要修改baidu_std_protocol里面New Message方法和Callback就可以了。还会有其他地方影响吗?

@jamesge
Copy link
Contributor Author

jamesge commented Dec 3, 2018

应该没有其他地方了。之前的难点是怎么能同时兼容pb 2.x和3.x,这对于大型repo是个刚需(否则一个程序中可能会链入两个版本的brpc)。但对于小项目,编译时选项是ok的。

@gethinzhang
Copy link

好,我们内部先改一下试试,多谢

@feng-y
Copy link

feng-y commented Dec 18, 2018

这里是每个pv自己申请一个arena, pb 的message从这个arena上分配,pv 结束后再回收这个arena么?

@gxkevin
Copy link

gxkevin commented Aug 2, 2019

这块目前有什么进展么,自己改的话需要改哪里 @gethinzhang

@huntinux
Copy link

huntinux commented May 19, 2021

尝试改了http协议里面的pb创建方式

diff --git a/src/brpc/policy/http_rpc_protocol.cpp b/src/brpc/policy/http_rpc_protocol.cpp
index 8787574..c5a498d 100644
--- a/src/brpc/policy/http_rpc_protocol.cpp
+++ b/src/brpc/policy/http_rpc_protocol.cpp
@@ -16,6 +16,7 @@
 //          Ge,Jun (gejun@baidu.com)
 
 #include <google/protobuf/descriptor.h>             // MethodDescriptor
+#include <google/protobuf/arena.h>             // MethodDescriptor
 #include <gflags/gflags.h>
 #include <json2pb/pb_to_json.h>                    // ProtoMessageToJson
 #include <json2pb/json_to_pb.h>                    // JsonToProtoMessage
@@ -71,6 +72,8 @@ DEFINE_bool(pb_enum_as_number, false, "[Not recommended] Convert enums in "
             "protobuf to json as numbers, affecting both client-side and "
             "server-side");
 
+DEFINE_bool(enable_pb_arena, false, "Create request/response message on arena");
+
 // Read user address from the header specified by -http_header_of_user_ip
 static bool GetUserAddressFromHeaderImpl(const HttpHeader& headers,
                                          butil::EndPoint* user_addr) {
@@ -685,6 +688,7 @@ public:
         : _cntl(std::move(s._cntl))
         , _req(std::move(s._req))
         , _res(std::move(s._res))
+        , _arena(std::move(s._arena))
         , _method_status(std::move(s._method_status))
         , _received_us(s._received_us)
         , _h2_stream_id(s._h2_stream_id) {
@@ -693,6 +697,7 @@ public:
 
     void own_request(google::protobuf::Message* req) { _req.reset(req); }
     void own_response(google::protobuf::Message* res) { _res.reset(res); }
+    void own_arena(google::protobuf::Arena* arena) { _arena.reset(arena); }
     void set_method_status(MethodStatus* ms) { _method_status = ms; }
     void set_received_us(int64_t t) { _received_us = t; }
     void set_h2_stream_id(int id) { _h2_stream_id = id; }
@@ -701,6 +706,7 @@ private:
     std::unique_ptr<Controller, LogErrorTextAndDelete> _cntl;
     std::unique_ptr<google::protobuf::Message> _req;
     std::unique_ptr<google::protobuf::Message> _res;
+    std::unique_ptr<google::protobuf::Arena> _arena;
     MethodStatus* _method_status;
     int64_t _received_us;
     int _h2_stream_id;
@@ -1424,10 +1430,19 @@ void ProcessHttpRequest(InputMessageBase *msg) {
     google::protobuf::Service* svc = sp->service;
     const google::protobuf::MethodDescriptor* method = sp->method;
     accessor.set_method(method);
-    google::protobuf::Message* req = svc->GetRequestPrototype(method).New();
-    resp_sender.own_request(req);
-    google::protobuf::Message* res = svc->GetResponsePrototype(method).New();
-    resp_sender.own_response(res);
+    google::protobuf::Message* req = nullptr;
+    google::protobuf::Message* res = nullptr;
+    if (FLAGS_enable_pb_arena) {
+      google::protobuf::Arena* arena = new google::protobuf::Arena();
+      req = svc->GetRequestPrototype(method).New(arena);
+      res = svc->GetResponsePrototype(method).New(arena);
+      resp_sender.own_arena(arena);
+    } else {
+      req = svc->GetRequestPrototype(method).New();
+      resp_sender.own_request(req);
+      res = svc->GetResponsePrototype(method).New();
+      resp_sender.own_response(res);
+    }
 
     if (__builtin_expect(!req || !res, 0)) {
         PLOG(FATAL) << "Fail to new req or res";

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement improvements on existing features official created by brpc authors
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants