Accumulator options?

[cheald]

I'm logging data from our applications to Telegraf via the UDP listener and the Influx line protocol, but it appears that Telegraf is only logging one data point per 10 second window - there's no aggregation of the values passed (counts, sums, etc). We might have hundreds of events per second, but only the most recent one is logged - all the others seem to get dropped. It's obviously desirable to not just write arbitrary numbers of datapoints into influx, but dropping data seems undesirable.

This functionality does exist in the statsd input plugin, but I was hoping to not use that so as to give consumers a much more flexible way to specify tags in the messages sent (rather than relying on statsd templating). Do we have any options besides the statsd plugin? Is there some way to configure a given input plugin to accumulate the values passed to it?

[sparrc]

The UDP listener should not be dropping any points, that sounds like a serious bug if it's really happening. What version of Telegraf are you using and can you give more details on what you're seeing?

Only the statsd input does counts, sums, etc. There is already a feature request to add that to telegraf more generally (#380)

FWIW our statsd listener does support tags: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/statsd#influx-statsd

[cheald]

$ telegraf -version
Telegraf - version 0.13.0

Here's a demo of what's going on:

https://gist.github.com/cheald/f21507116def1669a75017615812fa29

Basically, I can write 100 data points and confirm that they're all sent (via ncat), but when I send them to the UDP listener, only the final one ever makes it into influx. socket.rb is the producer, I test that the items are all sent by first listening with ncat, then I terminate ncat, start telegraf, run the producer again, and only one point makes it into influx. This is a totally stock Telegraf and Influx install, other than the udp_listener.conf addition.

I did go ahead and end up switching to the statsd listener and everything is working as expected now - I just wrote a custom statsd client that can speak Telegraf's augmented line protocol. So, the problem's solved for me at the moment, but the UDP listener's behavior surprised me.

My plan initially was to log timing data, then use count(column) via Grafana to get my event count, and I was very surprised to find that there was only 1 event per 10 seconds, rather than the hundreds expected. The addition of the extra columns provided by statsd is somewhat noisier than needed, but it does mean that the data gets rolled up a bit before being sent to influx, which is convenient for my use case.

[sparrc]

@cheald I see the problem. Since you are sending points without a timestamp to influxdb (which is configured by default for second precision), they are getting overwritten.

ie, each point gets the timestamp 1464861173000000000

after running your script as configured I get the same result:

> select * from test
name: test
----------
time            host    tag value
1464865524000000000 tyrion  foo 99

but if I change the influxdb precision to "ns", like this:

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
  ## The full HTTP or UDP endpoint URL for your InfluxDB instance.
  ## Multiple urls can be specified as part of the same cluster,
  ## this means that only ONE of the urls will be written to each interval.
  # urls = ["udp://localhost:8089"] # UDP endpoint example
  urls = ["http://localhost:8086"] # required
  ## The target database for metrics (telegraf will create it if not exists).
  database = "telegraf" # required
  ## Retention policy to write to.
  retention_policy = "default"
  ## Precision of writes, valid values are "ns", "us" (or "µs"), "ms", "s", "m", "h".
  ## note: using "s" precision greatly improves InfluxDB compression.
  precision = "ns"

then I get all of the points:

> select * from test
name: test
----------
time            host    tag value
1464865524000000000 tyrion  foo 99
1464865629847636574 tyrion  foo 0
1464865629847874684 tyrion  foo 1
1464865629847907630 tyrion  foo 2
1464865629847936351 tyrion  foo 3
1464865629847980941 tyrion  foo 4
1464865629848004885 tyrion  foo 5
1464865629848022755 tyrion  foo 6
1464865629848037720 tyrion  foo 7
1464865629848065661 tyrion  foo 8
1464865629848111970 tyrion  foo 9
1464865629848132159 tyrion  foo 10
1464865629848158140 tyrion  foo 11
1464865629848205254 tyrion  foo 12
1464865629848255689 tyrion  foo 13
1464865629848281912 tyrion  foo 14
1464865629848306402 tyrion  foo 15
1464865629848378775 tyrion  foo 16
1464865629848413340 tyrion  foo 17
1464865629848439594 tyrion  foo 18
1464865629848463560 tyrion  foo 19
1464865629848514941 tyrion  foo 20
1464865629848544994 tyrion  foo 21
1464865629848565654 tyrion  foo 22
1464865629848579599 tyrion  foo 23
1464865629848594781 tyrion  foo 24
1464865629848608870 tyrion  foo 25
1464865629848623999 tyrion  foo 26
1464865629848637779 tyrion  foo 27
1464865629848651220 tyrion  foo 28
1464865629848664909 tyrion  foo 29
1464865629848678439 tyrion  foo 30
1464865629848693550 tyrion  foo 31
1464865629848707229 tyrion  foo 32
1464865629848720431 tyrion  foo 33
1464865629848742322 tyrion  foo 34
1464865629848755215 tyrion  foo 35
1464865629848767893 tyrion  foo 36
1464865629848779361 tyrion  foo 37
1464865629848791665 tyrion  foo 38
1464865629848803069 tyrion  foo 39
1464865629848816730 tyrion  foo 40
1464865629848829892 tyrion  foo 41
1464865629848841018 tyrion  foo 42
1464865629848852265 tyrion  foo 43
1464865629848863601 tyrion  foo 44
1464865629848876342 tyrion  foo 45
1464865629848887257 tyrion  foo 46
1464865629848898698 tyrion  foo 47
1464865629848909842 tyrion  foo 48
1464865629848922755 tyrion  foo 49
1464865629848935738 tyrion  foo 50
1464865629848946840 tyrion  foo 51
1464865629848957949 tyrion  foo 52
1464865629848969250 tyrion  foo 53
1464865629848981710 tyrion  foo 54
1464865629848992705 tyrion  foo 55
1464865629849003697 tyrion  foo 56
1464865629849015973 tyrion  foo 57
1464865629849027099 tyrion  foo 58
1464865629849039279 tyrion  foo 59
1464865629849051823 tyrion  foo 60
1464865629849064382 tyrion  foo 61
1464865629849075521 tyrion  foo 62
1464865629849086627 tyrion  foo 63
1464865629849098795 tyrion  foo 64
1464865629849109928 tyrion  foo 65
1464865629849121124 tyrion  foo 66
1464865629849132018 tyrion  foo 67
1464865629849144409 tyrion  foo 68
1464865629849155599 tyrion  foo 69
1464865629849166426 tyrion  foo 70
1464865629849177454 tyrion  foo 71
1464865629849188442 tyrion  foo 72
1464865629849202219 tyrion  foo 73
1464865629849213568 tyrion  foo 74
1464865629849231266 tyrion  foo 75
1464865629849242808 tyrion  foo 76
1464865629849253802 tyrion  foo 77
1464865629849266053 tyrion  foo 78
1464865629849276955 tyrion  foo 79
1464865629849288022 tyrion  foo 80
1464865629849298928 tyrion  foo 81
1464865629849312902 tyrion  foo 82
1464865629849324065 tyrion  foo 83
1464865629849335130 tyrion  foo 84
1464865629849346159 tyrion  foo 85
1464865629849358733 tyrion  foo 86
1464865629849370942 tyrion  foo 87
1464865629849382029 tyrion  foo 88
1464865629849392938 tyrion  foo 89
1464865629849404002 tyrion  foo 90
1464865629849415098 tyrion  foo 91
1464865629849427226 tyrion  foo 92
1464865629849438386 tyrion  foo 93
1464865629849450368 tyrion  foo 94
1464865629849461165 tyrion  foo 95
1464865629849473326 tyrion  foo 96
1464865629849514674 tyrion  foo 97
1464865629849534662 tyrion  foo 98
1464865629849550875 tyrion  foo 99

[sparrc]

btw thanks for the great debugging information :), I'm closing this case because #380 already exists for requesting the generic aggregation feature.

[cheald]

Makes sense. Thank you!