Add config reload endoint / signal to promtail

[tomwilkie]

So we don't have to restart all the pods just to change the scrape config etc.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[hairyhenderson]

Seems like there's a PR for this pending - would be nice to see this happen instead of the issue going stale and disappearing 😉

[cyriltovena]

@hairyhenderson how this would be useful for you ? Can you give us more informations ?

[hairyhenderson]

@cyriltovena like the description says, it's annoying to restart processes to reload configs. I'm used to being able to send a signal or call a reload endpoint in Prometheus, and this is one area where Loki isn't "Like Prometheus, but for logs" 😉

[cyriltovena]

yeah but Prometheus could miss some scrape and it makes sense to not restart to avoid this kind of behaviour, promtail though won't miss anything as everything is on disk.

[hairyhenderson]

Sure... But there are side-effects of processes terminating which may be undesirable. And, it's conventional for processes in UNIX to listen to a SIGHUP and reload config based on that.

I suppose my question is: why would you not want this feature?

[pastatopf]

Upvote fromy my side :-)

[Perry-The-Cynic]

It's no longer true that (from promtail's perspective) "everything is on disk." The syslog endpoint has data pushed onto promtail, and depending on the retry policy of the sender restarting promtail may drop some messages.

[cyriltovena]

It's no longer true that (from promtail's perspective) "everything is on disk." The syslog endpoint has data pushed onto promtail, and depending on the retry policy of the sender restarting promtail may drop some messages.

Very good point. Although that seems easier to fix than implementing a reload.

[timurkhisamov]

Upvote

[tomasfreund]

Hi, this is a big issue for us. We provision client certificates for promtail with ACME dns-01 challenge and promtail does not seem to reload the certificates at all. We need promtail to either reload certificates or to have a config reload endpoint so that we can reuse existing tooling from Prometheus.

[marcusne]

Hi, same here /reload would be really helpful to maintain promtail in K8s or at least support e.g. SIGHUP

[liguozhong]

reload is a very dangerous feature, it is easy to make promtail panic, we can refer to the failure history of another log agent

● vectordotdev/vector#10485
● vectordotdev/vector#10412
● vectordotdev/vector#13228