storagenode,client: pipelining Append RPC

[ijsong]

Motivation

Currently, Append RPC is a simple request-response pattern in which a StorageNode responds with a reply message once a client calls Append RPC. It makes architecture simple but also provides total order in the log easily.

We can improve this RPC pattern by using pipelining; you can imagine HTTP pipelining.

Design

Bidirectional streaming Append RPC

To implement pipelined RPC, the current Append RPC, which is unary, is insufficient. The Append RPC has to support a bidirectional streaming RPC pattern. Fortunately, we can do that easily by using the gRPC feature.

Nonblocking API

Append RPC should be a nonblocking call to support pipelined Append RPC. It is also critical to keep order among sequential RPCs. That means the GLSNs of msg1, msg2, and msg3 must be ordered sequentially according to the invoking sequence.

client.Append(ctx, topicID, msg1)
client.Append(ctx, topicID, msg2)
client.Append(ctx, topicID, msg3)

sequenceDiagram
    Client->>StorageNode: AppendRequest(msg1)
    Client->>StorageNode: AppendRequest(msg2)
    Client->>StorageNode: AppendRequest(msg3)
    StorageNode->>Client: AppendResponse(msg1)
    StorageNode->>Client: AppendResponse(msg2)
    StorageNode->>Client: AppendResponse(msg3)

Loading

Challenges

Global Order

It must keep global sequential order between Append and AppendTo RPCs. However, it is not trivial in nonblocking calls.

Although the client invokes Append before AppendTo, the msg2's GLSN can be before msg1 since those Append-variant RPCs might add logs to different log streams.

client.Append(ctx, topicID, msg1)
client.AppendTo(ctx, topicID, types.LogStreamID(2), msg2)

sequenceDiagram
    Client->>StorageNode/LogStream1: AppendRequest(msg1)
    Client->>StorageNode/LogStream2: AppendRequest(msg2)
    StorageNode/LogStream2->>Client: AppendResponse(msg2)
    StorageNode/LogStream1->>Client: AppendResponse(msg1)

Loading