This repository contains the public API definitions for Liftbridge. It is primarily intended for Liftbridge client developers.
The client-facing gRPC API is defined in api.proto.
It is also possible for a client to publish messages to Liftbridge via NATS directly. Liftbridge accepts plain NATS messages, allowing it to make existing subjects durable without any publisher changes. However, these messages will not have some features such as acks.
In order to opt into Liftbridge-specific features, the message must be prefixed
with the following header and be encoded as a Message (defined in
api.proto).
All Liftbridge messages and RPCs sent over NATS are prefixed with an envelope header. This includes client-facing messages, such as publishes and acks, as well as internal RPCs like replication.
0 8 16 24 32
├───────────────┴───────────────┴───────────────┴───────────────┤
│ Magic Number │
├───────────────┬───────────────┬───────────────┬───────────────┤
│ Version │ HeaderLen │ Flags │ MsgType │
├───────────────┴───────────────┴───────────────┴───────────────┤
│ CRC-32C (optional) │
└───────────────────────────────────────────────────────────────┘
The Liftbridge magic number is B9 0E 43 B4. This was chosen by random but
deliberately restricted to invalid UTF-8 to reduce the chance of a collision.
This was also verified to not match known file signatures.
The version byte allows for future protocol upgrades. This should only be bumped if the envelope format changes or if the message encoding changes in a non-backwards-compatible way. Adding fields to the messages should not require a version bump.
Currently, the only supported protocol version is v0, i.e. 0x00.
The header length is the offset of the payload. This is included primarily for safety.
The flag bits are defined as follows:
| Bit | Description |
|---|---|
| 0 | CRC-32C enabled |
This is the Liftbridge-specific message type the envelope contains:
| MsgType | Name | Description | Internal |
|---|---|---|---|
| 0 | Publish | Client-published message | no |
| 1 | Ack | Server-published ack | no |
| 2 | ReplicationRequest | Request to replicate partition data | yes |
| 3 | ReplicationResponse | Response to ReplicationRequest | yes |
| 4 | RaftJoinRequest | Request to join Raft cluster | yes |
| 5 | RaftJoinResponse | Response to RaftJoinRequest | yes |
| 6 | LeaderEpochOffsetRequest | Request for partition leader's latest offset for epoch | yes |
| 7 | LeaderEpochOffsetResponse | Response to LeaderEpochOffsetRequest | yes |
| 8 | PropagatedRequest | Request forwarded to metadata leader | yes |
| 9 | PropagatedResponse | Response to PropagatedRequest | yes |
| 10 | ServerInfoRequest | Request for cluster information | yes |
| 11 | ServerInfoResponse | Response to ServerInfoRequest | yes |
| 12 | PartitionStatusRequest | Request to get partition status | yes |
| 13 | PartitionStatusResponse | Response to PartitionStatusRequest | yes |
| 14 | PartitionNotification | Signal new data is available for partition | yes |
The CRC-32C (Castagnoli) is the checksum of the payload (i.e. from HeaderLen to the end). This is optional but should significantly reduce the chance that a random NATS message is interpreted as a Liftbridge message.