[reliable payments] router payment state machine #2761

halseth · 2019-03-12T13:25:29Z

This PR introduces a persistent state machine to the ChannelRouter's payment flow, ensuring we can handle payment results that are received after a restart. It also paves they way for adding cancellation of payments, and resuming payment sessions across restarts.

Problem

lnd currently runs into problems if it is restarted while an HTLC is in flight on the network. The primary reason for this is the way the router hands the HTLC to the switch. The router persist no information about the payment, so when a result eventually comes back, we risk the information needed to properly populate the OutgoingPayment in the database is lost.

Solution

With the paymentStateMachine introduced in this PR, we store two pieces of key information:

The HTLC paymentID. This is used to query the Switch whether the HTLC is still active. If not active we know that we can safely retry the payment attempt. If active the router will wait for the result to be available, and store the result to the DB.
The HTLC's route. This is added to the DB together with the preimage when the payment succeeds.

Note

The Switch does not currently persist the pending payment attempts across restarts. This will be added in a follow-up PR.

Replaces #2475

Builds on

Roasbeef

I really dig this approach! With the state machine, I find it much easier to follow than the prior attempts at a solution to this problem. I've completed an initial pass so far, and the main question in my mind is the size of the overlap between this new state machine and the existing control tower in the switch. At one point the control tower addressed a need within the codebase, but it seems like this new state machine can eventually subsume the responsibilities of the control tower.

htlcswitch/switch.go

server.go

routing/router.go

htlcswitch/pending_payment.go

htlcswitch/switch.go

routing/route.go

routing/payment_state_machine.go

routing/router.go

joostjager

I am, as always, worried about the persistence and the work and inflexibility it may cause us in the future. Tried to mainly analyze this part of the PR.

Why don't you merge the switch pr first btw, work bottom up? Or will we have with just this PR already good working resumes of payments?

channeldb/payments.go

htlcswitch/pending_payment.go

routing/router.go

routing/payment_state_machine.go

joostjager · 2019-03-20T10:12:12Z

channeldb/payments.go

-		binary.BigEndian.PutUint64(paymentIDBytes, paymentID)
+// CompletePayment overwrites the OutgoingPayment stored in the DB for the
+// corresponding payment hash with the completed one.
+func (db *DB) CompletePayment(preimage lntypes.Preimage,


Looks unused

routing/router.go

routing/payment_state_machine.go

joostjager

Definitely nicer without that intermediate persistent state.

Main comments:

Code structure in router
Consolidation of payment related stores

channeldb/payments.go

routing/control_tower.go

routing/payment_store.go

routing/router.go

channeldb/payments.go

cfromknecht

nice refactor of the router/switch interactions, this should make the whole payment flow much tighter!

one question i have with new state introduced in the control tower, will we just treat all payments that are currently grounded as started but never attempted? an alternative would be to rename grounded payments as failed, and introduce a new state for initiated. not sure which is better atm

routing/router.go

cfromknecht · 2019-03-21T20:08:28Z

routing/router_test.go

+		return nil
+	}
+
+	ctx.router.cfg.GetPaymentResult = func(paymentID uint64) (


many of these GetPaymentResult funcs seem similar, any way we can generate the closures w/ less code duplication?

routing/router.go

routing/route_test.go

rpcserver.go

routing/router_test.go

routing/control_tower.go

htlcswitch/switch.go

channeldb/payments.go

routing/router.go

channeldb/control_tower.go

migrateOutgoingPayments moves the OutgoingPayments into a new bucket format where they all reside in a top-level bucket indexed by the payment hash. In this sub-bucket we store information relevant to this payment, such as the payment status. To avoid that the router resend payments that have the status InFlight (we cannot resume these payments for pre-migration payments) we delete those statuses, so only Completed payments remain in the new bucket structure.

Since we have performed a migration, the db should be in a consistent state, and we can remove the non-strict option.

This commit gives a new responsibility to the control tower, letting it populate the payment bucket structure as the payment goes through its different stages. The payment will transition states Grounded->InFlight->Success/Failed, where the CreationInfo/AttemptInfo/Preimage must be set accordingly. This will be the main driver for the router state machine.

This encapsulates all state needed to resume a payment from any point of the payment flow, and that must be shared between the different stages of the execution. This is done to prepare for breaking the send loop into smaller parts, and being able to resume the payment from any point from persistent state.

This commit makes the router use the ControlTower to drive the payment life cycle state machine, to keep track of active payments across restarts. This lets the router resume payments on startup, such that their final results can be handled and stored when ready.

On startup the router will fetch the in-flight payments from the control tower, and resume their execution.

TestRouterPaymentStateMachine tests that the router interacts as expected with the ControlTower during a payment lifecycle, such that it payment attempts are not sent twice to the switch, and results are handled after a restart.

And unexport deprecated code.

TestPaymentControlDeleteNonInFlight checks that calling DeletaPayments only deletes payments from the database that are not in-flight.

halseth added this to the 0.6 milestone Mar 12, 2019

halseth added the payments Related to invoices/payments label Mar 12, 2019

This was referenced Mar 12, 2019

[WIP] Router payments persistence #2475

Closed

[reliable payments] persist htlcswitch pending payments #2762

Merged

halseth force-pushed the reliable-payments-router-state-machine branch from e83fd93 to 2409e3d Compare March 12, 2019 13:55

Roasbeef requested changes Mar 15, 2019

View reviewed changes

halseth mentioned this pull request Mar 19, 2019

lnrpc+rpcserver: display payreq via listpayments #2717

Merged

halseth force-pushed the reliable-payments-router-state-machine branch 7 times, most recently from 78a0c81 to 1687cdc Compare March 20, 2019 10:22

joostjager reviewed Mar 20, 2019

View reviewed changes

halseth force-pushed the reliable-payments-router-state-machine branch 5 times, most recently from ce35b99 to 9d1bd42 Compare March 21, 2019 14:58

joostjager reviewed Mar 21, 2019

View reviewed changes

cfromknecht reviewed Mar 22, 2019

View reviewed changes

halseth force-pushed the reliable-payments-router-state-machine branch 3 times, most recently from 7180772 to 7aae12a Compare March 26, 2019 10:11

joostjager reviewed Mar 26, 2019

View reviewed changes

channeldb/payments.go Outdated Show resolved Hide resolved

routing/router.go Show resolved Hide resolved

joostjager reviewed Mar 26, 2019

View reviewed changes

channeldb/control_tower.go Outdated Show resolved Hide resolved

halseth force-pushed the reliable-payments-router-state-machine branch 2 times, most recently from fd20644 to ab0f77c Compare March 26, 2019 14:38

Roasbeef removed this from the 0.6 milestone Mar 26, 2019

halseth added 25 commits May 27, 2019 20:18

channeldb/control_tower: add FetchSentPayments

693807c

channeldb test: add TestOutgoingPaymentsMigration

b7189ba

channeldb/control_tower: remove non-strict option

bb4aadd

Since we have performed a migration, the db should be in a consistent state, and we can remove the non-strict option.

channeldb/payments: add StatusFailed

6d80661

channeldb/control_tower test: test new payment behavior

59c2557

routing/router: move sending and receiving payment result into loop

ae7bf2c

routing/payment_lifecycle: extract error handling into method

e9b2182

routing/payment_lifecycle: extract create attempt into method

dd73c51

routing/control_tower: add FetchInFlightPayments

677cb01

routing/router: resume payment state machine at startup

3323800

On startup the router will fetch the in-flight payments from the control tower, and resume their execution.

routing: export MissionControl

adc4640

routing/payment_session+router: make PaymentSession interface

f4306b1

routing+server: define PaymentSessionSource

3f76bc0

routing/router_test: add TestRouterPaymentStateMachine

60e2367

TestRouterPaymentStateMachine tests that the router interacts as expected with the ControlTower during a payment lifecycle, such that it payment attempts are not sent twice to the switch, and results are handled after a restart.

channeldb+router: record payment failure reason in db

1b78890

channeldb: move deprecated code to migration_09_legacy_serialization.go

d78d3f5

And unexport deprecated code.

channeldb+multi: rename Grounded->Unknown, Completed->Succeeded

d705b8a

channeldb: derive PaymentStatus implicitly from stored info

95b4828

control_tower: make initial bucket bucket creation stricter

589f0fc

routing/router_test: add test case for ForwardingError on SendHTLC

649ee87

channeldb/control_tower test: add TestPaymentControlDeleteNonInFlight

7cb25a5

TestPaymentControlDeleteNonInFlight checks that calling DeletaPayments only deletes payments from the database that are not in-flight.

halseth force-pushed the reliable-payments-router-state-machine branch from 9f7b1d9 to 7cb25a5 Compare May 27, 2019 18:19

Roasbeef merged commit 19fafd7 into lightningnetwork:master May 27, 2019

treygriffith mentioned this pull request Jun 8, 2019

Resolved hashes in payments do not reflect back their settled state #2032

Closed

Roasbeef mentioned this pull request Jul 2, 2019

[WIP] channeldb+routing+switch: move payment tracking from rpcserver to router+switch #2246

Closed

halseth mentioned this pull request Aug 21, 2020

Async payment cancellation #4562

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[reliable payments] router payment state machine #2761

[reliable payments] router payment state machine #2761

halseth commented Mar 12, 2019 •

edited by cfromknecht

Loading

Roasbeef left a comment

joostjager left a comment

joostjager Mar 20, 2019

joostjager left a comment

cfromknecht left a comment •

edited

Loading

cfromknecht Mar 21, 2019

[reliable payments] router payment state machine #2761

[reliable payments] router payment state machine #2761

Conversation

halseth commented Mar 12, 2019 • edited by cfromknecht Loading

Problem

Solution

Note

Roasbeef left a comment

Choose a reason for hiding this comment

joostjager left a comment

Choose a reason for hiding this comment

joostjager Mar 20, 2019

Choose a reason for hiding this comment

joostjager left a comment

Choose a reason for hiding this comment

cfromknecht left a comment • edited Loading

Choose a reason for hiding this comment

cfromknecht Mar 21, 2019

Choose a reason for hiding this comment

halseth commented Mar 12, 2019 •

edited by cfromknecht

Loading

cfromknecht left a comment •

edited

Loading