statsd-alerting-backend

A backend plugin for Statsd to perform some basic alerting on events.

Overview

Suppose you wanted to use StatsD as your main event stream aggregator due to its simple interface and its ease of integration with existing services. Perfect! Now, what if you want to send out some alerts based on certain events? Let's do it!

Configuring Alerts

There are two different types of alerts that are supported. Those are "event" alerts and "metric" alerts (see example configuration file for both). "Event" alerts are things that you want to be alerted on immediately. An example of an event you would likely want to be alerted on immediately is an uncaught exception.

"Metrics" alerts are a bit more complicated. StatsD publishes aggregate metrics at a configurable rate (default is every ten seconds). Perhaps you wanted to be alerted when the 90th percentile average time for an event exceeds a certain value. You can do that (check out the example configuration for how to set that up).

This plugin is very young and I plan to include more sophisticated alerting features as well as more alert types. Feel free to leave feedback via the issues for things you would like to see added.

Installation

In your StatsD installation folder, run:

$ npm install statsd-alerting-backend

Include the backend in your config.js file (see example configuration file below for complete configuration example).

{
  "backends": ["statsd-alerting-backend"]
}

Development

This plugin is written in CoffeeScript that is compiled to Javascript automatically when publishing to NPM (see gulpfile.js and package.json for more details). To work on this plugin, simply clone the repository and run npm install. I would suggest running gulp watch in a separate shell to watch the source Coffeescript files for changes and automatically compile them to Javascript files.

Configuration

dispatchers

Here you can define a set of dispatchers to use when sending alerts. They can be named anything (names must be unique as it is a Javascript object). See the example config file for formatting. Each dispatcher must have a type that is one of the valid alert dispatchers (currently "slack", "pagerduty", "hipchat", and "log"). Each dispatcher must also include the necessary configuration for the alert dispatcher.

Dispatcher Templates

You can specify a template to use for your alert messages if you'd like. If you don't specify a template, a sane default will be used for all alerts. Templates must be written using Handlebars.js (really only the basic templating features are useful here). An example template might look like:

{
  "template": "Event alert for {{name}} ({{metric}} {{type}})"
}

Templates are defined in each dispatcher's configuration block. You can specify a template for event alerts (using the template property) and for metric alerts (using the metricTemplate property). The context used when rendering templates is the event object itself. Each even contains a name, metric, type, and some other metadata.

slack configuration

List your Slack incoming webhook configuration information. Required keys are host and token. The username will default to "statsd-alerts" and the channel will default to #alerts.

pagerduty configuration

Simply list your Pagerduty service key.

sentry configuration

Simply list your Sentry DSN.

hipchat configuration

List your Hipchat API key and the room name to which you want to send the alerts. Optionally include a from description and a notify boolean.

Make sure that the key provided has enabled the scope send_notification.

log configuration

Target, one of stdout or stderr. Defaults to stdout.

email configuration

You must provide a transport key and a mailOptions key. See the nodemailer docs for specifics.

transport can be a connection string or an object, see examples here.

eventAlertSubject is the email subject for an event-based alert.

metricsEventAlertSubject is the email subject for a metric event-based alert.

mailOptions controls how emails are delivered. Valid keys include from and to. The subject is filled in using either eventAlertSubject or metricsEventAlertSubject. The text will be the formatted event or metrics event alert.

events

Think of this as a list of events (StatsD counters or gauges) that you want to send alerts immediately as they are sent. A good example of this would be an exception happening or a user signing up for something. You can use wildcards in the event name (see wildcard for formatting and matching information).

metrics

Think of this as a list of aggregate metrics you want to alert on. Typical uses for this would be alerting when an average time is greater than a set value or when the rate of counter exceeds some value. You could also alert when the difference (or delta) of a current metric and its previous value exceeds a certain value. See the integration test for more information on how to use these alerts. You can also use wildcards in the metrics name for matching. A wildcard will match any events of a specific type (i.e. timer_data or counter_rates).

Supported Alert Types

Currently you can alert using Slack, HipChat, Pagerduty, Sentry or logging to stdout. Please ensure that you have the proper configuration values for whichever alerting sources you specify in your configuration file.

Example Configuration

{
  backends: ["statsd-alerting-backend"],

  alerts: {
    dispatchers: {
      slackDispatcher: {
        type: "slack",
        config: {
          webhook: "<INCOMING_WEBHOOK>",
          username: "statsd-alerts",
          channel: "#alerts",
          template: "Saw a {{name}} event!"
        }
      },

      pagerdutyDispatcher: {
        type: "pagerduty",
        config: {
          key: "<PAGERDUTY_SERVICE_KEY>"
        }
      },

      sentryDispatcher: {
        type: "sentry",
        config: {
          dsn: "<SENTRY_DSN>"
        }
      },

      hipchatDispatcher: {
        type: "hipchat",
        config: {
          key: "<HIPCHAT_API_KEY>",
          from: "statsd-hipchat-alerts".
          room: "<HIPCHAT_ROOM_NAME>",
          color: "red",
          notify: true,
          template: "Saw a {{name}} event!"
        }
      },

      logDispatcher: {
        type: "log",
        config: {
          target: "stdout"
        }
      },

      emailDispatcher: {
        type: "email",
        config: {
          transport: "smtps://user%40gmail.com:pass@smtp.gmail.com",
          eventAlertSubject: "Oops! Event Alert",
          metricsEventAlertSubject: "Oops! Metrics Event Alert",
          mailOptions: {
            from: "user@gmail.com",
            to: "alerts@mycompany.com"
          }
        }
      }
    },

    events: [{
      name: "some.event.*",
      dispatcher: "logDispatcher"
    }],

    metrics: [{
      name: "some.*.timer",
      type: "timer_data",
      key: "mean_90",
      delta: 10,
      dispatcher: "slackDispatcher"
    }, {
      name: "some.event.counter",
      type: "counter_rates",
      gte: 0.2,
      dispatcher: "pagerdutyDispatcher"
    }]
  }
}

Contributing

Feel free to leave issues here or fork the project and submit pull requests. If there is a feature you would like added just submit an issue describing the feature and I will do my best.