Implement graceful shutdown with pipeline draining

[tigrannajaryan]

Currently when shutting down any data that is in the pipeline is lost. We need to drain the pipeline during shutdown. First we need to shutdown receivers, then processors (and implement draining in processors that stores data), then exporters.

[joe-elliott]

I may be misunderstanding the need, but except for the processors this appears to be done:

opentelemetry-collector/service/service.go

Lines 304 to 317 in b468dee

	func (app *Application) shutdownPipelines() {
	// Shutdown order is the reverse of building: first receivers, then flushing pipelines
	// giving senders a chance to send all their data. This may take time, the allowed
	// time should be part of configuration.
	app.logger.Info("Stopping receivers...")
	app.builtReceivers.StopAll()
	// TODO: shutdown processors by calling Shutdown() for each processor in the
	// order they are arranged in the pipeline.
	app.logger.Info("Shutting down exporters...")
	app.exporters.ShutdownAll()
	}

[tigrannajaryan]

except for the processors

Yes, this is the missing part. Processors support partially coming in #488

We still need to implement draining in processors (particularly in batch and queue).

[jan25]

Hi! Can i work on this?

After quick look at queued_processor, this is a rough idea to fix - On shutdown have a deadline of 5s or until queue is emptied(whichever happens first). Also, We can have stats on how many spans were flushed and dropped during shutdown process. WDYT?

I see there is Stop() in queued_processor, any hints where is that used?

Similar idea can be applied to batch_processor too.

[tigrannajaryan]

@jan25 do you still want to work on this?

We will need to drain the pipeline by first stopping receivers, then stopping processors in the right order, then stop exporters. When stopping we will need to wait for up to certain max time, you are right. It needs to be done for all components, not just certain processors.

[jan25]

@tigrannajaryan Sure, I'll start the work and open a PR soon

[tigrannajaryan]

@jan25 I will remove this from 1.0 milestone, since it is not a must-have, but feel free to work on it. We will accept it when it is ready.

[kamalmarhubi]

@tigrannajaryan Is there a suggested workaround to avoid dropping messages during collector upgrades?

[tigrannajaryan]

There is no known workaround.

[kamalmarhubi]

@tigrannajaryan in that case, I feel like this should be added back to the GA milestone? Dropping things on the floor during upgrades is not great...

[tigrannajaryan]

@kamalmarhubi Help is wanted on this one. If there are contributors who can submit a PR before GA, we will be happy to review it.

[bogdandrutu]

I think this is done correct?

[clintonb]

Is graceful shutdown actually implemented/supported? The docs say it is not:

opentelemetry-collector/CONTRIBUTING.md

Lines 493 to 500 in ba4ab80

	### Graceful Shutdown
	Collector does not yet support graceful shutdown but we plan to add it. All components
	must be ready to shutdown gracefully via `Shutdown()` function that all component
	interfaces require. If components contain any temporary data they need to process
	and export it out of the Collector before shutdown is completed. The shutdown process
	will have a maximum allowed duration so put a limit on how long your shutdown
	operation can take.

[bradmac-ats]

Is graceful shutdown actually implemented/supported? The docs say it is not:

opentelemetry-collector/CONTRIBUTING.md

Lines 493 to 500 in ba4ab80

	### Graceful Shutdown
	Collector does not yet support graceful shutdown but we plan to add it. All components
	must be ready to shutdown gracefully via `Shutdown()` function that all component
	interfaces require. If components contain any temporary data they need to process
	and export it out of the Collector before shutdown is completed. The shutdown process
	will have a maximum allowed duration so put a limit on how long your shutdown
	operation can take.

These docs seem to indicate that it has been implemented: https://pkg.go.dev/go.opentelemetry.io/collector/otelcol#CollectorSettings