Skip to content

Latest commit

 

History

History
103 lines (71 loc) · 3.13 KB

iobuf.md

File metadata and controls

103 lines (71 loc) · 3.13 KB
Raw

中文版

brpc uses butil::IOBuf as data structure for attachment in some protocols and HTTP body. It's a non-contiguous zero-copied buffer, proved in previous projects, and good at performance. The interface of IOBuf is similar to std::string, but not the same.

If you've used the BufHandle in Kylin before, you should notice the convenience of IOBuf: the former one is badly encapsulated, leaving the internal structure directly in front of users, who must carefully handle the referential countings, very error prone and leading to bugs.

What IOBuf can:

  • Default constructor does not allocate memory.
  • Copyable. Modifications to the copy doesn't affect the original one. Copy the managing structure of IOBuf only rather the payload.
  • Append another IOBuf without copying payload.
  • Can append string, by copying payload.
  • Read from or write into file descriptors.
  • Serialize to or parse from protobuf messages.
  • constructible like a std::ostream using IOBufBuilder.

What IOBuf can't:

  • Used as universal string-like structure in the program. Lifetime of IOBuf should be short, to prevent the referentially counted blocks(8K each) in IOBuf lock too many memory.

Cut

Cut 16 bytes from front-side of source_buf and append to dest_buf:

source_buf.cut(&dest_buf, 16); // cut all bytes of source_buf when its length < 16

Just pop 16 bytes from front-side of source_buf:

source_buf.pop_front(16); // Empty source_buf when its length < 16

Append

Append another IOBuf to back-side：

buf.append(another_buf);  // no data copy

Append std::string to back-sie

buf.append(str);  // copy data of str into buf

Parse

Parse a protobuf message from the IOBuf

IOBufAsZeroCopyInputStream wrapper(&iobuf);
pb_message.ParseFromZeroCopyStream(&wrapper);

Parse IOBuf in user-defined formats

IOBufAsZeroCopyInputStream wrapper(&iobuf);
CodedInputStream coded_stream(&wrapper);
coded_stream.ReadLittleEndian32(&value);
...

Serialize

Serialize a protobuf message into the IOBuf

IOBufAsZeroCopyOutputStream wrapper(&iobuf);
pb_message.SerializeToZeroCopyStream(&wrapper);

Built IOBuf with printable data

IOBufBuilder os;
os << "anything can be sent to std::ostream";
os.buf();  // IOBuf

Print

Directly printable to std::ostream. Note that the iobuf in following example should only contain printable characters.

std::cout << iobuf << std::endl;
// or
std::string str = iobuf.to_string(); // note: allocating memory
printf("%s\n", str.c_str());

Performance

IOBuf is good at performance:

Action Throughput QPS
Read from file -> Cut 12+16 bytes -> Copy -> Merge into another buffer ->Write to /dev/null 240.423MB/s 8586535
Read from file -> Cut 12+128 bytes -> Copy-> Merge into another buffer ->Write to /dev/null 790.022MB/s 5643014
Read from file -> Cut 12+1024 bytes -> Copy-> Merge into another buffer ->Write to /dev/null 1519.99MB/s 1467171