Cvcal #5
Cvcal #5
Conversation
Conflicts: app/kafka/manager/ActorModel.scala app/kafka/manager/KafkaManager.scala conf/routes test/kafka/manager/TestKafkaManager.scala
Update sbt, play, scala versions
Add logkafka - Collect logs and send lines to Apache Kafka
…s to the topics. The KafkaStateActor now tracks the following additional information: - adds a Curator treeCache to cache the consumer information, which includes the names of consumer instances that own each partition, and the offsets at every partition. - adds cached offsets for the topics. These must be gotten directly from Kafka as they are not stored in Zookeeper, so the call functions by using a simple consumer for every partition. Currently, the frequency of updates to these offsets is hard-coded to every 5 seconds or less frequently. - configurable in the clusterConfig is the possibility to exclude any consumer that do not have an offsets/ owners/ AND ids/ subdirectory as being "inactive" The UI now has a consumer list view, which lists all consumers and the topics they consume from, an individual page for each consumer that lists each topic with the cumulative lag and the percentage of partitions that have an owner (if all is well, this should be 100%), a page for every consumed topic that lists the offset, lag, and owner information at every partition for that consumer-topic combination. To the topic views, there is now the most recent offset at every partition and the cumulative log size, as well as links to the consumed state of the topic inside a list of the consumers that consume from this topic.
|
Hi @cvcal, I'm looking into getting us up-to-date with yahoo's master branch now, primarily so we can make use of the topic-based consumer offset managment in kafka 8.2. From the history it looks like you may have gotten an equivalent pull request into yahoo from your own account - is that the case? |
|
Yeah, I'm surprised you're only updating it now. When I left, the pull request hadn't made it in, so I left this fork up instead. I haven't followed what has changed on yahoo's side, but I assume the functionality Knewton wanted is still there in some form or other. You should feel comfortable syncing to the latest release. |
|
Great, thanks! |
@patelh fixed the merge conflicts so that yahoo's kafka-manager can accept pull request yahoo#108. Please merge this!