Restart delay not working when agent process exits unexpectedly

[andykellr]

Component(s)

cmd/opampsupervisor

What happened?

Description

When the supervisor fails to start the collector, it will log that it will restart the collector in a bit, but it doesn't actually wait 5s to restart and instead logs the message over and over.

Steps to Reproduce

Send a bad config from an OpAMP server to the supervisor. The collector will fail to start and exit. The supervisor will be caught in a tight loop logging that it will "restart in a bit..."

Expected Result

The supervisor will wait 5s and then attempt to restart the collector.

Actual Result

The supervisor logs this message over and over.

2023-10-18T21:51:28.581-0400	DEBUG	supervisor/supervisor.go:619	Agent process exited unexpectedly. Will restart in a bit...	{"pid": 46996, "exit_code": 1}
2023-10-18T21:51:28.581-0400	DEBUG	supervisor/supervisor.go:619	Agent process exited unexpectedly. Will restart in a bit...	{"pid": 46996, "exit_code": 1}
2023-10-18T21:51:28.581-0400	DEBUG	supervisor/supervisor.go:619	Agent process exited unexpectedly. Will restart in a bit...	{"pid": 46996, "exit_code": 1}
2023-10-18T21:51:28.581-0400	DEBUG	supervisor/supervisor.go:619	Agent process exited unexpectedly. Will restart in a bit...	{"pid": 46996, "exit_code": 1}
2023-10-18T21:51:28.581-0400	DEBUG	supervisor/supervisor.go:619	Agent process exited unexpectedly. Will restart in a bit...	{"pid": 46996, "exit_code": 1}
2023-10-18T21:51:28.581-0400	DEBUG	supervisor/supervisor.go:619	Agent process exited unexpectedly. Will restart in a bit...	{"pid": 46996, "exit_code": 1}
2023-10-18T21:51:28.581-0400	DEBUG	supervisor/supervisor.go:619	Agent process exited unexpectedly. Will restart in a bit...	{"pid": 46996, "exit_code": 1}
2023-10-18T21:51:28.581-0400	DEBUG	supervisor/supervisor.go:619	Agent process exited unexpectedly. Will restart in a bit...	{"pid": 46996, "exit_code": 1}
2023-10-18T21:51:28.581-0400	DEBUG	supervisor/supervisor.go:619	Agent process exited unexpectedly. Will restart in a bit...	{"pid": 46996, "exit_code": 1}
2023-10-18T21:51:28.581-0400	DEBUG	supervisor/supervisor.go:619	Agent process exited unexpectedly. Will restart in a bit...	{"pid": 46996, "exit_code": 1}

Collector version

05ec3a2

Environment information

Environment

OS: m1 Mac
Compiler(if manually compiled): go1.21.3 darwin/arm64

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

Based on a quick review of the code, when the Commander watch() method closes the doneCh, the Supervisor gets stuck in case <-s.commander.Done() as the closed channel will continue to produce signals.

Tested using the OpAMP Agent Extension PR
#16594

[github-actions]

Pinging code owners:

• cmd/opampsupervisor: @evan-bradley @atoulme @tigrannajaryan

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• cmd/opampsupervisor: @evan-bradley @atoulme @tigrannajaryan

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• cmd/opampsupervisor: @evan-bradley @atoulme @tigrannajaryan

See Adding Labels via Comments if you do not have permissions to add labels yourself.