Snap publisher plugin - Kafka

Allows publishing of data to Apache Kafka

It's used in the snap framework.

1. Getting Started

• System Requirements
• Installation
• Configuration and Usage

2. Documentation

• Kafka Quickstart
• Published data
• Examples
• Roadmap

3. Community Support
4. Contributing
5. License
6. Acknowledgements

Getting Started

System Requirements

• Uses sarama golang client for Kafka by Shopify

Installation

Download Kafka plugin binary:

You can get the pre-built binaries for your OS and architecture at plugin's GitHub Releases page.

To build the plugin binary:

Fork https://github.com/intelsdi-x/snap-plugin-publisher-kafka

Clone repo into $GOPATH/src/github.com/intelsdi-x/:

$ git clone https://github.com/<yourGithubID>/snap-plugin-publisher-kafka.git

Build the plugin by running make within the cloned repo:

$ make

This builds the plugin in ./build

Configuration and Usage

• Set up the snap framework

Task Manifest Config

In task manifest, in config section of Kafka publisher the following settings can be declared:

Key	Type	Default value	Description
topic	string	"snap"	The topic to send messages
brokers	string	"localhost:9092"	Semicolon delimited list of "server:port" brokers Kafka's standard broker communication port is 9092

Documentation

Kafka Quickstart

This is a minimal-configuration needed to run the Kafka broker service on Docker

Run ZooKeeper server on docker

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one.

 $ docker run -d --name zookeeper jplock/zookeeper:3.4.6

Check if ZooKeeper docker is running:

$ docker ps  

	CONTAINER ID        IMAGE                      COMMAND                CREATED             STATUS              PORTS              			NAMES
	9b0ddbdd75cd        jplock/zookeeper:3.4.6     "/opt/zookeeper/bin/   38 seconds ago      Up 38 seconds       2181/tcp, 2888/tcp, 3888/tcp	zookeeper

Run Kafka server on docker

 docker run -d --name kafka --link zookeeper:zookeeper ches/kafka

Verify the running docker containers:

$ docker ps

	CONTAINER ID        IMAGE                        COMMAND                CREATED             STATUS              PORTS							NAMES
	dfb3cdfb3f87        ches/kafka:latest            "/start.sh"            7 seconds ago       Up 6 seconds        7203/tcp, 9092/tcp				kafka
	9b0ddbdd75cd        jplock/zookeeper:3.4.6       "/opt/zookeeper/bin/   3 minutes ago       Up 3 minutes        2181/tcp, 2888/tcp, 3888/tcp	zookeeper

Get Kafka advertised hostname:

$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' kafka

  172.17.0.14

Read more about Kafka on http://kafka.apache.org

Published data

The plugin publishes all collected metrics serialized as JSON to Kafka. An example of published data is below:

[
  {
    "timestamp": "2016-07-25T11:27:59.795548989+02:00",
    "namespace": "/intel/mock/bar",
    "data": 82,
    "unit": "",
    "tags": {
      "plugin_running_on": "my-machine"
    },
    "version": 0,
    "last_advertised_time": "2016-07-25T11:27:21.852064032+02:00"
  },
  {
    "timestamp": "2016-07-25T11:27:59.795549268+02:00",
    "namespace": "/intel/mock/foo",
    "data": 72,
    "unit": "",
    "tags": {
      "plugin_running_on": "my-machine"
    },
    "version": 0,
    "last_advertised_time": "2016-07-25T11:27:21.852063228+02:00"
  }
]

Examples

Example of running psutil collector plugin and publishing data to Kafka.

Set up the Snap framework

Ensure Snap daemon is running:

• initd: service snap-telemetry start
• systemd: systemctl start snap-telemetry
• command line: sudo snapteld -l 1 -t 0 &

Download and load Snap plugins (paths to binary files for Linux/amd64):

$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-kafka/latest/linux/x86_64/snap-plugin-publisher-kafka
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-psutil/latest/linux/x86_64/snap-plugin-collector-psutil
$ snaptel plugin load snap-plugin-publisher-kafka
$ snaptel plugin load snap-plugin-collector-psutil

Create a task manifest (see exemplary tasks), for example psutil-kafka.json with following content:

{
  "version": 1,
  "schedule": {
    "type": "simple",
    "interval": "1s"
  },
  "workflow": {
    "collect": {
      "metrics": {
        "/intel/psutil/load/load1": {},
        "/intel/psutil/load/load15": {}
      },
      "publish": [
        {
          "plugin_name": "kafka",
          "config": {
            "topic": "test",
            "brokers": "172.17.0.14:9092"
          }
        }
      ]
    }
  }
}

Create a task:

$ snaptel task create -t psutil-kafka.json

Watch created task:

$ snaptel task watch <task_id>

To stop previously created task:

$ snaptel task stop <task_id>

Roadmap

There isn't a current roadmap for this plugin, but it is in active development. As we launch this plugin, we do not have any outstanding requirements for the next release.

If you have a feature request, please add it as an issue and/or submit a pull request.

Community Support

This repository is one of many plugins in Snap, a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap To reach out to other users, head to the main framework

Contributing

We love contributions!

There's more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.

License

Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.

Acknowledgements

• Author: Nicholas Weaver

And thank you! Your contribution, through code and participation, is incredibly important to us.