NewServerConfiguration() called twice after config reload with -watch causing e.g. double health probes/jobs
#692
Comments
With func realmain(ctx context.Context, arguments []string) int {
...
case _, more := <-reloadCh:
...
_, reloadErr = runtime.NewServerConfiguration(cf, logger.WithFields(fields), tmpMemStore)
...
execCmd, restartSignal = newRestartableCommand(ctx, cmd) // replace previous pair
go func() {
// logger settings update gets ignored at this point
// have to be locked for an update, skip this feature for now
errCh <- execCmd.Execute(args, confFile, logger)
}()and func (r *Run) Execute(args Args, config *config.Couper, logEntry *logrus.Entry) error {
...
srvConf, err := runtime.NewServerConfiguration(config, logEntry, memStore) |
johakoch
changed the title
-watchNewServerConfiguration() called twice after config reload with -watch causing e.g. double health probes
|
Jobs are started twice after reload, too. |
johakoch
changed the title
NewServerConfiguration() called twice after config reload with -watch causing e.g. double health probesNewServerConfiguration() called twice after config reload with -watch causing e.g. double health probes/jobs
|
We have a context key called We should add this condition to all related objects like |
|
The But there are 2 calls to reloadCh := watchConfigFiles(confFile.Files, logger, flags.FileWatchRetries, flags.FileWatchRetryDelay)
for {
select {
...
case _, more := <-reloadCh:
...
restartSignal <- struct{}{} // shutdown running couper
<-errCh // drain current error due to cancel and ensure closed ports
execCmd, restartSignal = newRestartableCommand(ctx, cmd) // replace previous pair
go func() {
// logger settings update gets ignored at this point
// have to be locked for an update, skip this feature for now
errCh <- execCmd.Execute(args, confFile, logger)
^^^^^^^^^^^^^^^
}()The other is caused in response to an error. case err = <-errCh:
if err != nil {
if netErr, ok := err.(*net.OpError); ok {
if netErr.Op == "listen" && errRetries < flags.FileWatchRetries {
errRetries++
logger.Errorf("retry %d/%d due to listen error: %v", errRetries, flags.FileWatchRetries, netErr)
// configuration load succeeded at this point, just restart the command
execCmd, restartSignal = newRestartableCommand(ctx, cmd) // replace previous pair
time.Sleep(flags.FileWatchRetryDelay)
go func() {
errCh <- execCmd.Execute(args, confFile, logger)
^^^^^^^^^^^^^^^
}()
continue
... |
|
I guess, the first of the two mentioned calls to time.Sleep(flags.FileWatchRetryDelay)(or similar) ahead of the call, there's no retry error and so only one probe running after a reload. |
|
However, even if the retry is triggered differently, there should only be one probe. |
With
-watch, after a config reload,Example:
Start Couper
Normal health check:
Reload config:
Double health check log:
Health probe shutdown log:
Still double health check logs:
The text was updated successfully, but these errors were encountered: