Removes the instance database record when provision fails

[bnaecker]

In the case that instance provision fails in the final saga node, the
actual booting of the Propolis zone, there was previously a no-op undo
action. This leaves the instance record in the database, perpetually in
a "starting" state. It can't be moved out of that state, because that
requires a full instance-ensure request to the sled-agent, which tries
that last action again, which fails, and ...

This adds an actual undo action, which sets the instance state in the
database to failed, and then deletes. That state change is needed
because we can't delete instances in the "starting" state. State changes
are normally only made in response to the sled agent observing a state
change in the actual instance, but is valid in this case since there
is no such instance.

[bnaecker]

It's not really easy to write an integration test for this, for a few reasons. I was however able to run tests using the Oxide CLI, which show that the record has in fact been removed.

bnaecker@feldspar : ~/cli $ cargo run -- instance list -o o -p p
    Finished dev [unoptimized + debuginfo] target(s) in 0.14s
     Running `target/debug/oxide instance list -o o -p p`
 id | name | description | hostname | memory | ncpus | project_id | run_state | time_created | time_modified | time_run_state_updated |
----+------+-------------+----------+--------+-------+------------+-----------+--------------+---------------+------------------------

bnaecker@feldspar : ~/cli $ cargo run -- instance create -o o -p p vm0
    Finished dev [unoptimized + debuginfo] target(s) in 0.13s
     Running `target/debug/oxide instance create -o o -p p vm0`
instance description:: vm0
instance hostname:: vm0
instance memory:: 4
instance ncpus:: 2
✘ Oxide API internal error: Internal Server Error
bnaecker@feldspar : ~/cli $ cargo run -- instance list -o o -p p
    Finished dev [unoptimized + debuginfo] target(s) in 0.15s
     Running `target/debug/oxide instance list -o o -p p`
 id | name | description | hostname | memory | ncpus | project_id | run_state | time_created | time_modified | time_run_state_updated |
----+------+-------------+----------+--------+-------+------------+-----------+--------------+---------------+------------------------

[smklein]

Github isn't making commenting on this specific line easy for me, but...

... shouldn't the sic_delete_instance_record undo action in CreateInstanceRecord be responsible for actually doing the deletion of the instance?

I bring this up because having an undo action for the last operation in a saga seems a little funny - my mental model for actions is that they should "do one thing in the state graph (or not, and fail)". This means that the last action generally either "happens or it doesn't", and usually there isn't anyone who can call "undo" on it.

Anyway - we mentioned this in chat, but figured I'd post it here for visibility

[bnaecker]

For further context -- I had basically fixed this earlier, but not rebased the branch I was working on on top of that fix. That's in #810, for the record.

[bnaecker]

Abandoning.