server: start htlcswitch early in the pipeline

[yyforyongyu]

After a recent change, a consistent failure has been showing in the test revoked_uncooperative_close_retribution_zero_value_remote_output from the build with backend=bitcoind dbbackend=postgres, lnd node was stuck at starting channel arbitrator. Here's what could happen.

Suppose a node starts with a channel breached with a state StateContractClosed,

1. chain arbitrator starts -> channel arbitrator starts
2. when starting channel arbitrator, we advance the channel state
3. in that advanceState, when we begin with StateContractClosed, it will send msg to htlcswitch via c.cfg.DeliverResolutionMsg
4. htlcswitch hasn't started yet, causing the above function never returns, and the node will be stuck

This PR fixes it by starting htlcswitch early. By checking the dependencies of htlcswitch.Start(), it makes a few db queries and must be started after Notifier since it needs to subscribe new blocks. There's also a callsite on s.chanRouter via htlcswitch.Start() -> s.ForwardPackets -> s.cfg.FetchLastChannelUpdate -> s.chanRouter.GetChannelByID. However I think this doesn't require s.chanRouter to be started since it's just querying the db.

To end the starting dependecy issue, I think we need to rely on checking states from disk and not from other subservers.

[Crypt-iQ]

Pretty sure we can also fix #2998 at the same time by handling StateContractClosed explicitly here:

lnd/contractcourt/channel_arbitrator.go

Lines 465 to 470 in 60625b6

	switch c.state {
	case StateDefault:
	fallthrough
	case StateBroadcastCommit:
	fallthrough
	case StateCommitmentBroadcasted:

edit: can confirm the above suggestion with this pr fixes the resolutionmsg issue. this is because if the proper trigger is set, the HtlcFailNowActions get recalculated (usually they are not for the chainTrigger), so even if lnd goes down during the hand-off, the hand-off will occur again on start-up and the switch can handle duplicates

[Crypt-iQ]

My previous comment is a little wrong: this won't fully fix #2998 but my suggestion does make it better. The switch doesn't persist them, so if they are held in a mailbox after the hand-off and lnd crashes, the switch won't get them again. The hand-off could be changed so that they're fully handled before returning but out of scope here.

[yyforyongyu]

Pretty sure we can also fix #2998 at the same time by handling StateContractClosed explicitly here:

You mean we just fallthrough here for StateContractClosed?

[Crypt-iQ]

Pretty sure we can also fix #2998 at the same time by handling StateContractClosed explicitly here:

You mean we just fallthrough here for StateContractClosed?

I think it can be left to another pr that fixes the issue i mentioned

[Crypt-iQ]

nit: third commit message could be changed, but code is good

[lightninglabs-deploy]

@Roasbeef: review reminder

[Roasbeef]

LGTM 👑