Add backoff to logging of bad subscriptions

[gunnaraasen]

Bug report

Steps to reproduce:

1. With a fresh local InfluxDB install, create a test database.
2. Start a local kapacitord process, which will automatically generate a subscription for the test database.
3. Write a few points to the test database.
4. Stop Kapacitor.
5. Write more points to the test database.

Actual behavior: Every write to a database generates a log line for each subscription which fails to deliver points to it's destination. This quickly results in an overwhelming number of logs along the lines of:

[subscriber] 2016/08/16 11:33:22 Post http://localhost:9092/write?consistency=&db=test&precision=ns&rp=autogen: dial tcp [::1]:9092: getsockopt: connection refused

Expected behavior: The database should back off logging of write failures to a subscriber and/or disable/drop a subscription which fails to accept a certain number of writes.

The subscriber measurement in the _internal database already records pointsWritten and writeFailures so logging the write failure is only needed for diagnosing connection errors.

This problem is particularly bad when a Kapacitor instance is removed, resulting in orphaned subscriptions that are tedious to clean up and continuously spam the logs on every write.

This is relevant to discussions in issues #5462 and #7036.

[nathanielc]

Are we planning on adding retry logic with the backoff logic? If not then the backoff doesn't seem all that useful. If so then we should design that out as it will require a significant amount of RAM.

This problem is particularly bad when a Kapacitor instance is removed, resulting in orphaned subscriptions that are tedious to clean up and continuously spam the logs on every write.

Slightly unrelated but, we should make cleaning up subscriptions easier as backoff logic is not a solution to that problem. Adding syntax to drop all subscriptions with the same name, would make removing all Kapacitor related subscriptions easy.

[gunnaraasen]

I made this issue specifically to discuss logging, which I now realize was not entirely clear. The discussion around back-off/retry for the subscription points is happening on #5462.

Logging is an issue because each Kapacitor will create a large number of separate subscriptions and each subscription will log an error for every timeout. This can quickly lead to a significant amount of logging on every subscription timeout. Logging only the initial subscription timeout would help significantly cut down the amount of logs.

I think cleaning up subscriptions would definitely help make it easier to avoid a bunch of orphaned subscriptions in the first place.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please reopen if this issue is still important to you. Thank you for your contributions.