Waypoint server panics when runner is forgotten before stopped

[demophoon]

Describe the bug
When a Waypoint runner is forgotten before it is stopped, when that runner is stopped the Waypoint server panics.

I believe this happens because when the runner is forgotten, the runner record is removed from boltdb. Once the Waypoint runner is stopped the server notices and attempts to mark the runner as offline. During that process, server attempts to fetch the runner from BoltDB.

waypoint/internal/server/boltdbstate/runner.go

Lines 304 to 312 in 822d3ca

	func (s *State) runnerOffline(dbTxn *bolt.Tx, memTxn *memdb.Txn, id string) error {
	r, err := s.runnerById(dbTxn, id)
	if status.Code(err) == codes.NotFound {
	r = nil
	err = nil
	}
	if err != nil {
	return err
	}

If either r is set to nil or codes.NotFound is the error that is returned, r is nil by the time we attempt to determine what Kind of runner we are dealing with.

waypoint/internal/server/boltdbstate/runner.go

Line 322 in 822d3ca

switch r.Kind.(type) {

In this r = nil case the server panics and exits.

Steps to Reproduce
Steps to reproduce the behavior.

1. Start a Waypoint server
2. Start a preadopted Waypoint runner

export WAYPOINT_SERVER_ADDR=localhost:9701
export WAYPOINT_SERVER_TLS=true
export WAYPOINT_SERVER_TLS_SKIP_VERIFY=true
export WAYPOINT_SERVER_TOKEN=$(waypoint user token) # May need to be two commands
waypoint runner agent -id=testrunner

3. Forget the Waypoint runner in a new terminal

waypoint runner forget testrunner

4. Stop the Waypoint runner

At this point the server will have panicked.

Expected behavior
The server should continue as if the runner had always been forgotten in the first place.

Waypoint Platform Versions

• Waypoint CLI Version: v0.8.1
• Waypoint Server Platform and Version: docker v0.8.2, but non-platform specific

[briancain]

It'll be useful to audit the runner code path for runners when a runner is forgotten/deleted from bolt. There are likely bugs here since the adoption flow is recently new, so there are probably places that we need to fix up some bad behaviors. edit: It might also be worth to see if there's any runner logic to extract into something more generic outside of the boltdb server implementation.