Cloud Run Graceful Termination Support

[jonahrosenblum]

Is your feature request related to a problem? Please describe.

Google Cloud Run is implementing a new feature to support graceful termination, ideally open telemetry would also support this. In fact, this has already been requested before #523

Describe the solution you'd like

Add process event handlers which will listen for SIGTERM signals sent to the process and initiate the shutdown method, similar to how this has been implemented in Python.

Describe alternatives you've considered

The previous issue suggested something to do with callbacks to ensure all data is processed. I am not sure if concerns over thread blocking are an issue with the main solution I am suggesting.

Additional context

I am thinking of using the nodejs process library for detecting the SIGTERM, and a testing strategy which either involves process forking and/or mocking signals. I would appreciate feedback on the best testing/implementation strategies!

[dyladan]

Does #1296 solve your issue?

[jonahrosenblum]

I do not think so? I may be misunderstanding what this PR fixes but would this change cause shutdown to be automatically initiated if the process receives a SIGTERM? This is how this feature has been implemented in Python.

[dyladan]

ah no. Ok I misunderstood what you were asking for.

[markwolff]

Is the Cloud Run using signals to end the process? Is the beforeExit or exit event not sufficient for catching its termination? To me, I would expect data loss when force killing a process, e.g ctrl+c (SIGINT?).

[jonahrosenblum]

Currently processed are SIGKILL'ed when an instance is scaled down, the new Cloud Run will send a SIGTERM and allow for a grace period before a SIGKILL is sent. In this period processes should send any data which has not been exported yet. So yes, I believe signals are being used. I do not know much about beforeExit and exit but I assume they would not work in this instance?

[jonahrosenblum]

I forgot to ask earlier but could I be assigned to this issue?

[dyladan]

done

[jonahrosenblum]

@dyladan I believe this issue can be closed now?