pkg/reloader: improve detection of directory changes (#3136)

* pkg/reloader: improve detection of directory changes

When watching for changes in directories, the reloader used to rely only
on the watch interval and not the inotify events. This commit implements
a more efficient detection of changes for watched directories.

The change also adds a new `DelayInterval` option that allows to delay
the config reload after no additional event are received.

Finally a new metric,
`thanos_sidecar_reloader_config_apply_operations_total`, is added and
`thanos_sidecar_reloader_config_apply_errors_total` has been renamed
to `thanos_sidecar_reloader_config_apply_operations_failed_total` for
consistency.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Updates after Kemal's review

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

CHANGELOG.md

@@ -11,6 +11,10 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
 
 ## Unreleased
 
+### Changed
+
+- [#3136](https://github.com/thanos-io/thanos/pull/3136) Sidecar: Add metric `thanos_sidecar_reloader_config_apply_operations_total` and rename metric `thanos_sidecar_reloader_config_apply_errors_total` to `thanos_sidecar_reloader_config_apply_operations_failed_total`.
+
 ## [v0.15.0](https://github.com/thanos-io/thanos/releases) - 2020.09.07
 
 Highlights:

pkg/reloader/example_test.go

@@ -27,6 +27,7 @@ func ExampleReloader() {
 		WatchedDirs:   []string{"/path/to/dirs"},
 		WatchInterval: 3 * time.Minute,
 		RetryInterval: 5 * time.Second,
+		DelayInterval: 1 * time.Second,
 	})
 
 	ctx, cancel := context.WithCancel(context.Background())