Apply error in loop should not kill the sidecar

[mneverov]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Fixes: #2873

Log error when applying changes in the prometheus config instead of stopping the sidecar.
Add new metric reloader_config_errors_total to reflect config readings errors.
Add new flags --reloader.retry-interval and --reloader.watch-interval to tweak corresponding configs for the reloader.

NOTE
I found prometheus.ready_timeout flag with underscore, which seems to be a typo. Please, let me know if it should be changed to prometheus.ready-timeout in this PR (breaking change?).

Verification

To verify it manually add the following flags to the quickstart.sh:

    --reloader.config-file=data/prom"${i}"/prometheus.yml \
    --reloader.retry-interval=5s \
    --reloader.watch-interval=15s

After start, remove/rename data/prom0/prometheus.yml.
The sidecar should still run, the error is logged component=reloader msg="apply error" err="hash file: open data/prom0/prometheus.yml: no such file or directory".
Check metrics: the new metric reloader_config_errors_total is updated.

[bwplotka]

I think it looks good for me, but I would add some metric.

PTAL @s-urbaniak @paulfantom from Prometheus Operator perspective (you use config reloader with this package 🤔 )

[bwplotka]

Can we have metric for that as well? (:

[kakkoyun]

Just to give a heads up @ebabani is also trying fix the same problem #2952

[bwplotka]

This is not the same problem I believe

[kakkoyun]

Yeah, my mistake not the same problem but the solution is somewhat converging something similar.

[bwplotka]

Amazing, solid code. Just one minor suggestion for metric name to be more explicit! 💪

[bwplotka]

Can we also remove WIP prefix from PR title?

[mneverov]

@bwplotka applied the changes. Please take a look 🙏
The prometheus.ready_timeout flag will not be addressed here, correct?

[bwplotka]

Awesome 👍

LGTM, thanks!