Add performance metrics to tasks and node

[nathanielc]

Fixes #215

This PR exposes two metrics to the user about throughput and execution time:

1. The throughput in batches or points per second per taks is calculated depending on task type
2. The average execution time per node is calculated.

These values are exposed via the Kapacitor show command and in the internal statistics Kapacitor emits. As a result it is possible to graph throughput and execution times or other stats a node exposes.

[yosiat]

Hi @nathanielc ,
I am just wondering why put the metrics values and key concatenated in the label attribute and don't create new attribute for each metric?
For example:

Instead of:

digraph testTaskName {
graph [label="Throughput: 0.00 points/s"];
stream0 [label="stream0 0"];
stream0 -> stream1 [label="0"];
stream1 [label="stream1 0"];
}

Do something like this:

digraph testTaskName {
graph [throughput="0.00"];
stream0 [processed="0"];
stream0 -> stream1 [processed="0"];
stream1 [processed="0"];
}

• The throughput will be a well known format - "points per second"
• I have omitted the "stream0" / "stream1" since I don't know what they mean.

This way the DOT graph will be much easier to parse and show in a UI for example and it's valid syntax (DOT Attributes)

[nathanielc]

Both locally and via g.gravizo.com those attributes are ignored. They need to be label attributes. I am open to suggestions to make it better but it needs to remain valid/useful DOT syntax.

![Alt text](http://g.gravizo.com/g?
digraph testTaskName {
graph [label="Throughput: 1634.25 points/s"];
stream0 [label="stream0 154ns"];
stream0 -> stream1 [label="156467"];
stream1 [label="stream1 1.25ms"];
})

vs

![Alt text](http://g.gravizo.com/g?
digraph testTaskName {
graph [throughput="1634.25"];
stream0 [exectime="154ns"];
stream0 -> stream1 [processed="156467"];
stream1 [exectime="1.25ms"];
})

[yosiat]

You want it to be valid for view and for parsing, so let's merge our solutions: both label and attributes.

[nathanielc]

@yosiat I think you are right, but the labels will clutter it up too much. I think by default I will do it your way as that is more human readable which is the common use case. And then a flag can be specified to use labels instead so it renders correctly. Thoughts?

I don't like the complexity of something that should be so simple :(

Maybe there is another graph language that is easier to read and still renders well? I'll take a look around.

[nathanielc]

Looking through all of these https://gephi.org/users/supported-graph-formats/ I still like DOT the best as it reads well. For now I guess we can add the flag and revisit later if it becomes cumbersome to set the flag when its needed.

[yosiat]

@nathanielc I agree with this option of flags.
About graph formats - if you don't need the "renderabillity" you can use your own json format.

By the way - is is possible to see all that metrics in InfluxDB & Chronograf?

Self answeing myself here, from the release notes:

The internal statistics includes a new measurement named node that contains any stats a node provides, tagged by the task, node, task type and kind of node (i.e. window vs union).