AMQP auto reconnect feature

[ekini]

I can't find find the code in telegraf, which detects if servers from output plugins go away. It runs Connect() only once.

So if rabbitmq server gets down, telegraf repeatedly logs "can't flush points".
This commit fixes that using func (*Connection) NotifyClose and reconnecting after Close event.

I'm open to suggestions if there's a better way to do it.

[sparrc]

I'm not sure I like having a thread just for this output sink running to keep the connection open. Instead, could you modify the flush function (https://github.com/influxdb/telegraf/blob/master/agent.go#L300-L318) to retry making a connection if Write fails once?

[ekini]

But it will add unnecessary logic in the core. I'd rather leave it in the output pugin.
The thread doesn't run constantly, most of the time it just waits for a close event from amqp.

[ekini]

Although, if you/we are going to fix #187, it might be worth implementing connect/reconnect feature for all outputs altogether.

[sparrc]

@ekini I don't think it's unnecessary logic, I actually think that it's a necessary improvement that we'll need to make, I think most long-running telegraf instances will lose connection to one or more output sinks at some point

[ekini]

Ok, I'm going to close this one to prepare for a more common solution, suitable for all outputs.

[sparrc]

@ekini sorry for the delay getting to this.

After thinking more about it, I think that your solution may in fact be the best option. Many of the outputs (such as InfluxDB) do not manage a persistent connection like AMQP does. In the case of AMQP, it makes sense, and it also makes sense to make management of those connections on a per-output basis.

I'm going to re-open the PR, do you mind rebasing and I can this merged tomorrow during the USA/Pacfic workday?

[ekini]

Ok, rebased to current master.