collectd plugin not receives as fast.

[liyichao]

(May related to #2428) Recently, I setup collectd + influxdb 0.9, where every host has a collectd client and sends data to a central collectd server. The server then proxy to infuxdb's collectd plugin in another host.

But I notice that there are many points missing when query influxdb.

time                value
2015-05-09T10:45:23.925567Z 2.52
2015-05-09T11:02:43.925573Z 2.76
2015-05-09T11:15:53.925591Z 2.84
2015-05-09T11:21:13.925569Z 2.72
2015-05-09T11:21:43.925569Z 2.69
2015-05-09T11:24:13.925612Z 2.68
2015-05-09T11:26:33.92557Z  2.54
2015-05-09T11:32:43.92557Z  2.68
2015-05-09T11:34:13.925575Z 2.75
2015-05-09T11:35:43.925571Z 2.94

where each point should appear every 10s.

I guess udp may be the problem, so I find out udp packet drops:

➜  ~  netstat -su
IcmpMsg:
    InType0: 1
    InType3: 127
    OutType3: 767
    OutType8: 1
Udp:
    3943492 packets received
    163319 packets to unknown port received.
    1947095 packet receive errors
    4070491 packets sent
    RcvbufErrors: 1947095

while ethool -S shows no rx_discards or rx_errors.

Then I adjust my sysctl:

➜  ~  sysctl -n net.core.rmem_default
16777216
➜  ~  sysctl -n net.core.rmem_max
16777216

But influxdb seems not catching up. Then I glance at the implementation of the collectd plugin, and find that all things: read from udp, parse the message, write to influxdb are done in a single goroutine, so I guess it's the problem of this plugin, am I right? I am willing to provide more information if needed.

By the way, we flush data once every 10s in collectd client, and server is set with network plugin which forwards to another influxdb 0.9.0rc29 server. And below is udp packet drop in my collectd server.

➜  etc  netstat -su
IcmpMsg:
    InType0: 32
    InType3: 2128
    InType8: 37
    OutType0: 36
    OutType3: 3288
    OutType8: 268
Udp:
    798541820 packets received
    2180 packets to unknown port received.
    4259 packet receive errors
    32523056 packets sent
    RcvbufErrors: 4259

[otoolep]

Thanks @liyichao -- this is a dupe of #2428.