storm-metrics-influxdb

Table of Contents

1. Overview
2. Description - What it does and why it is useful
3. Setup - The basics of getting started with it
   • Last Version and Repository
4. Usage - Configuration options and additional functionality
   • Example
5. Reference - An under-the-hood peek at what the module is doing and how
6. Limitations - OS compatibility, etc.
7. Development - Guide for contributing to the module
8. Authors - Who created it
9. Collaborators - Who is contributing to do it
10. License

Overview

This is an Apache Storm Metrics Consumer that forward its metrics to InfluxDB to be used in real-time. Then you can use Grafana to make a Dashboard with your metrics.

Description

Apache Storm has a powerful metrics engine as you can see in Apache Storm Metrics, and we can implement our Own Metrics Consumer. This is what exactly it does, it implements a Metrics Consumer that send all of Metrics collected by Apache Storm to InfluxDB.

Setup

Version

The actual version of artifacts are:

Releases artifacts

Maven repository: Maven repository

Server: Maven release repository

<dependency>
    <groupId>com.github.christiangda</groupId>
    <artifactId>storm-metrics-influxdb</artifactId>
    <version>See release version in the link upper</version>
</dependency>

SNAPSHOT artifacts

Server: Maven snaphot repository

<dependency>
  <groupId>com.github.christiangda</groupId>
    <artifactId>storm-metrics-influxdb</artifactId>
    <version>See release version in the link upper</version>
</dependency>

Usage

Applying Metrics Collector to a specific topology

...
    topologyConf.registerMetricsConsumer(com.github.christiangda.storm.metrics.InfluxDBMetricsConsumer.class, 1);
    topologyConf.put("metrics.reporter.name", "InfluxDBMetricsConsumer");
    topologyConf.put("metrics.influxdb.url", "<http://YOUR_INFLUXDB_HOSTNAME:PORT>");
    topologyConf.put("metrics.influxdb.username", "<YOUR_INFLUXDB_USERNAME>");
    topologyConf.put("metrics.influxdb.password", "<YOUR_INFLUXDB_PASSWORD>");
    topologyConf.put("metrics.influxdb.database", "<YOUR_INFLUXDB_DATABASE>");
    topologyConf.put("metrics.influxdb.measurement.prefix", "<YOUR_INFLUXDB_MEASUREMENT_PREFIX>");
    topologyConf.put("metrics.influxdb.enable.gzip", <true or false>);
...

Applying Metrics Collector to all Apache Storm Cluster

topology.metrics.consumer.register:
  - class: "com.github.christiangda.storm.metrics.InfluxDBMetricsConsumer"
    parallelism.hint: 1
    argument:
      metrics.reporter.name: "com.github.christiangda.storm.metrics.InfluxDBMetricsConsumer"
      metrics.influxdb.url: "http://YOUR_INFLUXDB_HOSTNAME:PORT"
      metrics.influxdb.username: "YOUR_INFLUXDB_USERNAME"
      metrics.influxdb.password: "YOUR_INFLUXDB_PASSWORD"
      metrics.influxdb.database: "YOUR_INFLUXDB_DATABASE"
      metrics.influxdb.measurement.prefix: "YOUR_INFLUXDB_MEASUREMENT_PREFIX"
      metrics.influxdb.enable.gzip: true

Example

I've created a Maven Apache Storm project to show you an example, if you want to view it go to test-storm-metrics-influxdb

Reference

• Apache Storm
• Apache Storm Metrics Consumer
• Apache Storm Configuration
• InfluxDB
• InfluxDB Java API
• Grafana

Limitations

• Apache Storm >= 1.0.0
• InfluxDB >= 1.0.0
• Java >= 8

Development

• Fork it / Clone it (git clone https://github.com/christiangda/storm-metrics-influxdb.git; cd storm-metrics-influxdb)
• Create your feature branch (git checkout -b my-new-feature)
• Install Maven
• Make your changes / improvements / fixes / etc, and of course your Unit Test for new code
• Run the tests (mvn clean compile test package)
• Commit your changes (git add . && git commit -m 'Added some feature')
• Push to the branch (git push -u origin my-new-feature)
• Create new Pull Request

Of course, bug reports and suggestions for improvements are always welcome.

You can also support my work on powerdns via

This project use Maven Release Plugin to do easy the process of deploy to Github repository and Maven Repository

Authors

• Christian González

Collaborators

• Nikolay Melnikov
• Stanislaw Stefanowski

License

This module is released under the Apache License Version 2.0:

• http://www.apache.org/licenses/LICENSE-2.0.html