-
Notifications
You must be signed in to change notification settings - Fork 70
Core Concepts
A log stores entries or records, usually all sharing the same format and encoding characteristics. Entries can only be appended to a log, they cannot be inserted between other existing older entries. Read more in Jey Krep's write-up.
Topics are categories of messages produced by producer applications, and consumed by other applications. For example "notifications", "clicks", "orders", "index updates".
A partition is the physical division of a topic. Partitions are used for redundancy (when not running in stand-alone mode, they are replicated to other brokers in a Tank cluster), and also facilitate partitioning work among consumers, whereas each partition is drained(consumed from) a single consumer. Also, since each partition must fit on the servers that host it, using multiple partitions allows the topic to scale beyond the size of a single node.
Messages within a partition are totally ordered (no ordering guarantees across different partitions).
A message is made up of a value(arbitrary data, of upto 4GB in size, though typically it should be much lower than that), a timestamp in milliseconds that represents the time the message was created(event time, as opposed to processing time which is when a consumer - e.g a data processing engine receives the event from TANK), and, optionally, a key of length upto 255 characters for that message. Messages usually don't have keys, but they can be useful for compactions and stream processing topologies.
Compaction is implemented and supported but not yet enabled on Tank. It will be tunable via configuration options. Stream processing will be implemented in future releases.
Every message is identified by a sequence number.
When an application produces messages to Tank, it sends a request with 1+ messages. Those messages are all stored together in a bundle. A bundle holds a set of messages. By operating on bundles, Tank can consider compression for all messages in the bundle -- multiple messages increase the likelihood for a higher compression ratio, whereas, for example, bundles with only one message, unless that message is fairly long, will likely not be suitable for compression. The Tank Client uses heuristics to select a compression codec based on the set of messages, and is responsible for decompressing bundles received in consume responses.
Every message is uniquely identified by a sequence number, a uint64-bit integer. The first message published into a partition is assigned sequence number 1. Every subsequent message published is assigned a sequence number equal to the last assigned number plus 1. The sequence number is a per-partition monotonically increasing number (analogous to Kafka's "offsets").
All bundles(which include messages) are stored physically in segments on each broker's underlying filesystem. A segment is comprised of a log file and an index file. There is a 'current' segment for each topic, and its append-only log and index files are mutable, where all older 'sealed/frozen' past segments(log and index file pairs) are immutable.
Based on retention and limits configuration, the current segment may become an immutable segment and added to the list of other tracked immutable segments, and a new mutable segment will be created and become current.
There will always be a single mutable segment. Their log file will be named as num_ts.log and their index file as num.index, whereas num is the sequence number of the first message stored in that segment, and ts is the creation timestamp in Unix epoch time.
There will be 0 or more immutable segments. Their log file will be named as first-last_ts.ilog and their index file as first.index, whereas first is the sequence number of the first message stored in that segment, and, conversely, last sequence number of the last message stored in the segment. ts is the creation timestamp in Unix epoch time.