Actions Added Post Shutdown

[PeterEFinch]

There is a question of what should happen when an action is added after Shutdown has been called or a signal has been received.

---

Example scenario:
A microservice is starting up and initialising databases, HTTP/gRPC servers, etc. Part way through this initialising process an interrupt signal is received.

Current approach:
Once the interrupt signal has been received all the actions that have been added will be called in the prescribed order. Actions added after this will be ignored. It is this way because the order is usually set to first in, last done (first instantiated, last torn down) and there is no way to ensure this order is followed after the signal has been received meaning it will likely not be graceful anyway.

There are two different situations to be considered:

• An action is added while the shutdown actions previously added are being performed.
• An action is added after all the shutdown actions previously added have been performed.

A few different questions come to mind - the answer may differ depending on the situation being considered.

1. Should the actions be performed or not?
2. Should the actions be performed immediately or added to a queue of other actions being performed while following the order as closely as possible.
3. Should adding the actions be a blocking method which waits until the actions have been performed.
4. Should this behaviour be fixed or an option?
5. Does this matter?

---

This example scenario can be demonstrated using the main below. In this case the main will never finish.

package main
​
import (
	"context"
	"fmt"
	"os"
	"syscall"
	"time"
​
	"github.com/Graphmasters/safedown"
)
​
func main() {
	sa := safedown.NewShutdownActions(safedown.FirstInLastDone, syscall.SIGTERM)
​
	// Pretend database
	ctxA, cancelA := context.WithCancel(context.Background())
	sa.AddActions(cancelA)
​
	// This mimics the service receiving a termination signal
	process := os.Process{Pid: os.Getpid()}
	if err := process.Signal(syscall.SIGTERM); err != nil {
		fmt.Printf("error sending signal: %s", err)
	}
	time.Sleep(time.Second) // Time to ensure the signal reached safedown and shutdown was completed.
	
	// Pretend HTTP server
	ctxB, cancelB := context.WithCancel(context.Background())
	sa.AddActions(cancelB)
​
	<-ctxB.Done() // This waits for the pretend HTTP server to be closed
	<-ctxA.Done() // This waits for the pretend database to be close
}

[PeterEFinch]

Initialising the services inside go routines prevents the problem of the service never stopping. It it also works well for adding new HTTP, gRPC servers etc. Shutdown won't be graceful if a signal is sent during the initialisation period but the service will fail fast.

package main
​
import (
	"context"
	"fmt"
	"os"
	"syscall"
	"time"
​
	"github.com/Graphmasters/safedown"
)
​
func main() {
	sa := safedown.NewShutdownActions(safedown.FirstInLastDone, syscall.SIGTERM)
	defer sa.Wait()
​
	go initialisePretendServer(sa)
	go initialiseAnotherPretendServer(sa)
}
​
func initialisePretendServer(sa *safedown.ShutdownActions) {
	// Pretend database
	ctxA, cancelA := context.WithCancel(context.Background())
	sa.AddActions(cancelA)
​
	// This mimics the service receiving a termination signal
	process := os.Process{Pid: os.Getpid()}
	if err := process.Signal(syscall.SIGTERM); err != nil {
		fmt.Printf("error sending signal: %s", err)
	}
	time.Sleep(time.Second) // Time to ensure the signal reached safedown and shutdown was completed.
​
	// Pretend HTTP server
	ctxB, cancelB := context.WithCancel(context.Background())
	sa.AddActions(cancelB)
​
	<-ctxB.Done() // This waits for the pretend HTTP server to be closed
	<-ctxA.Done() // This waits for the pretend database to be close
	sa.Shutdown()
}
​
func initialiseAnotherPretendServer(sa *safedown.ShutdownActions) {
	// Another pretend server
	ctxA, cancelA := context.WithCancel(context.Background())
	sa.AddActions(cancelA)
​
	<-ctxA.Done() // Wait till the other pretend server is finished
	sa.Shutdown()
}

[PeterEFinch]

One alternative approach is to make breaking changes and change the signature of the AddActions so that it returns something, such as a boolean or an error:

func (sa *ShutdownActions) AddActions(...func()) bool
func (sa *ShutdownActions) AddActions(...func()) error

However, this requires a lot more checking:

func main() {
	sa := safedown.NewShutdownActions(safedown.FirstInLastDone, syscall.SIGTERM)
​
	// Pretend database
	ctxA, cancelA := context.WithCancel(context.Background())
	if !sa.AddActions(cancelA) {
		return
	}
​
	// This mimics the service receiving a termination signal
	process := os.Process{Pid: os.Getpid()}
	if err := process.Signal(syscall.SIGTERM); err != nil {
		fmt.Printf("error sending signal: %s", err)
	}
	time.Sleep(time.Second) // Time to ensure the signal reached safedown and shutdown was completed.
	
	// Pretend HTTP server
	ctxB, cancelB := context.WithCancel(context.Background())
​	if !sa.AddActions(cancelB) {
		return
	}

	<-ctxB.Done() // This waits for the pretend HTTP server to be closed
	<-ctxA.Done() // This waits for the pretend database to be close
}