[bug]: missing settled invoices (gap in settle_index)

[C-Otto]

Background

The "settle_index" is supposed to be monotonically increasing:

The "settle" index of this invoice. Each newly settled invoice will increment this index making it monotonically increasing. Callers to the SubscribeInvoices call can use this to instantly get notified of all settled invoices with an settle_index greater than this one. Note: Output only, don't specify for creating an invoice.

However, my node does not have any invoice with settle_index 30278 nor 50310, whereas all other invoices exist according to
lncli listinvoices --index_offset 0 --paginate-forwards --max_invoices 3000000 | jq -r .invoices[].settle_index.

The settled invoices following the gaps were settled on 2022-02-14 02:59:10+01 and 2022-06-05 22:10:46+02.

Your environment

• lnd v0.15.3-beta (with a few unrelated patches)
• Linux server 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
• bitcoind v23

[positiveblue]

Possible AMP invoices?

Could you check the amp_invoice_state field for lncli listinvoices --index_offset 0 --paginate-forwards --max_invoices 3000000 | jq '.invoices[] | select(.is_amp == true)'?

[C-Otto]

I get four results, all of them with "amp_invoice_state": {} (settle_index is 3,000-8,000 for these)

[positiveblue]

I checked on my own db and I had two missing. When I checked I was able to find "settle_index": "XX" and it was inside an amp_invoice. For AMP invoices settle_index is left to 0 in the invoice obj and it is set in the amp_invoice_state

{
  "memo": "just an amp invoice",
  "r_preimage": null,
  "r_hash": "...",
  ...
  "settled": true,
  "settle_date": "0",
  "add_index": "YYY",
  "settle_index": "0",
  "state": "SETTLED",
  "htlcs": [
    {
      "chan_id": "chan_id",
      "state": "SETTLED",
      "amp": {
        "set_id": "ZZZ",
        ...
      }
    }
  ],
  "is_amp": true,
  "amp_invoice_state": {
    "ZZZ": {
      "state": "SETTLED",
      "settle_index": "XXX",
      "settle_time": "...",
    }
  }
}

For every amp invoice paid you should have a gap in the settle_index from the jq command that you sent.

Do you know what versions (approx) you were running when the gap occurred?

[C-Otto]

2022-02-08 15:41:24.247 [INF] LTND: Version: 0.14.1-beta commit=v0.14.1-beta-5-g7f121f20b, build=production, logging=default, debuglevel=info,CNCT=debug,DISC=warn,HSWC=debug,NANN=debug,WTCL=warn

2022-05-26 12:02:27.384 [INF] LTND: Version: 0.15.0-beta.rc3 commit=v0.15.0-beta.rc3-6-g9aef65858, build=production, logging=default, debuglevel=info,CNCT=debug,DISC=warn,HSWC=debug,NANN=debug,WTCL=warn

[Roasbeef]

If you've ever deleted a settled invoice, then you might have a gap, see this fragment: https://github.com/lightningnetwork/lnd/blob/master/channeldb/invoices.go#L3384-L3404

[Roasbeef]

The "settle_index" is supposed to be monotonically increasing:

This is the case. But if invoice are ever deleted, then there'll be a gap there. If the numbers decreased or duplicated then it would no longer be monotonically increasing.

[C-Otto]

@Roasbeef I don't see any code that would delete a settled invoice. How would I do that? If I ever did that, would that appear in my logs (I still have them)?

And thank you, I just realized that the gap doesn't violate the documented behavior. However, if gaps are allowed, this makes it much harder to know that all settled invoices have been processed (i.e. that no pending invoice might settle in the future and fill the gap).

[C-Otto]

Aside from that, shouldn't it be "strictly monotonically increasing", i.e. no two invoices share the same settle_index?

[GeorgeTsagk]

Aside from that, shouldn't it be "strictly monotonically increasing", i.e. no two invoices share the same settle_index?

I believe that this wouldn't be feasible if a gap is ever created. You can't enforce strictness if there is also the possibility of a gap creation. As in order to fill the gap you'd need to break the monotonic increase.

I assume the purpose of settle_index is a quick way of getting invoices that settled before/after an invoice whose settle_index you have knowledge of? Thus no need for strictness or having all values populated, only order matters.

[C-Otto]

1 2 4 5 ... is (strictly) monotonically increasing, and 1 2 2 3 3 ... is monotonically increasing (i.e., never decreasing). The gaps are irrelevant.

Aside from math and definitions, my problem remains:
How can I deal with gaps, i.e. how can I make sure that I've seen all settled invoices there will ever be (up to some offset)? How can I avoid re-checking every gap, because it might have been filled a second ago?