"ERROR: apply message failed" upon paych voucher best-spendable query after voucher submission

[ghost]

Describe the bug
Under certain conditions, paych voucher best-spendable <channel id> gives the error message "ERROR: apply message failed: voucher has an outdated nonce, cannot redeem (RetCode=16)"

Generally, this is happening on a payment channel where the "best spendable" voucher has just been submitted successfully. However, there are other vouchers in the channel, so I believe paych voucher best-spendable should not just error out like this.

This bug is either just a low-severity "confusing error message" (if submitting the best-spendable is intended to clear away all other spendable vouchers) or it's a more serious blocker to redeeming a valid voucher (if negation of the previous parenthetical).

To Reproduce
The reproduction steps require two machines (called "to" and "from"). "from" should have a secp256k1 wallet with >= 1.0 FIL and will be the payer of funds, and "to" should have a BLS wallet (balance doesn't matter) and will be the receiver of funds.

Note: in the commands below, to $ means the prompt on the "to" machine; from $ means the from machine.

I create the payment channel like this:

from $ lotus paych get t14hsid4nwp724r6dxwgue5kvvzjqjrd6y4lalecq t3schhtef3m5yshlop3ffwjk5lauaqryx2e67xayngbj5stvha6m6i5noqvtiswykqg5pel4wt5j6zlrbblxbq 5000000000
t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a

Then I immediately created two vouchers for (1e9 and 2e9 attofil, respectively):

from $ lotus paych voucher create t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a 1000000000
i1UC5p1RJZfUi1pfGcoDNwPXTRTlg0IAAED2AAJFADuaygAAgFhCAYdpWxcpIUos5ufI4MDLqvjV5SHwKuOOEfB-uaDeqIJHPtnh2A9dQ3qQL7GsDQxMDP8MX0PNy9F6RxoL0AqCQT4A
from $ lotus paych voucher create t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a 2000000000
i1UC5p1RJZfUi1pfGcoDNwPXTRTlg0IAAED2AANFAHc1lAAAgFhCAa-myCUZEEs-0UrsaTn7nLe6a-uDrvMJwOrc-Y0X8SJwB9XKaO3piTI1N37WEg1N-SfGHCt2gjy6qr37GJkR0fIB

List vouchers for the channel:

from $ lotus paych voucher list t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a
Lane 0, Nonce 1: 1150
Lane 0, Nonce 2: 1000000000
Lane 0, Nonce 3: 2000000000

On "to" machine: check the current best spendable, manually add the 1000000000 attofil voucher, and verify it is the new best-spendable:

$ ./lotus paych voucher best-spendable t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a
i1UC5p1RJZfUi1pfGcoDNwPXTRTlg0IAAED2AAFDAAR-AIBYQgFg4MDyo8VGDivv6QmawB9ngXHIISZpVGXDqe2mWennVXusw5A0spST1HEffEsE0Ls_keXddgXf5X3FuurCEDv3AQ
Amount: 1150
to $ ./lotus paych voucher add t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a i1UC5p1RJZfUi1pfGcoDNwPXTRTlg0IAAED2AAJFADuaygAAgFhCAYdpWxcpIUos5ufI4MDLqvjV5SHwKuOOEfB-uaDeqIJHPtnh2A9dQ3qQL7GsDQxMDP8MX0PNy9F6RxoL0AqCQT4A
$ ./lotus paych voucher best-spendable t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a
i1UC5p1RJZfUi1pfGcoDNwPXTRTlg0IAAED2AAJFADuaygAAgFhCAYdpWxcpIUos5ufI4MDLqvjV5SHwKuOOEfB-uaDeqIJHPtnh2A9dQ3qQL7GsDQxMDP8MX0PNy9F6RxoL0AqCQT4A
Amount: 1000000000

State of the “to” machine right BEFORE voucher submission:

to $ ./lotus wallet balance
50976.230659335644412602 FIL
to $ ./lotus paych voucher best-spendable t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a
i1UC5p1RJZfUi1pfGcoDNwPXTRTlg0IAAED2AAJFADuaygAAgFhCAYdpWxcpIUos5ufI4MDLqvjV5SHwKuOOEfB-uaDeqIJHPtnh2A9dQ3qQL7GsDQxMDP8MX0PNy9F6RxoL0AqCQT4A
Amount: 1000000000

Here’s the submit command:

to $ ./lotus paych voucher submit t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a i1UC5p1RJZfUi1pfGcoDNwPXTRTlg0IAAED2AAJFADuaygAAgFhCAYdpWxcpIUos5ufI4MDLqvjV5SHwKuOOEfB-uaDeqIJHPtnh2A9dQ3qQL7GsDQxMDP8MX0PNy9F6RxoL0AqCQT4A
channel updated successfully

State of “to” machine 15 MINUTES AFTER voucher submission completed:

to $ ./lotus wallet balance
50976.230659335635546829 FIL

to $ ./lotus paych voucher best-spendable t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a
ERROR: apply message failed: voucher has an outdated nonce, cannot redeem (RetCode=16)

^^^ This is where the error occurs.

Expected behavior
I expect lotus paych voucher best-spendable to list the best remaining spendable voucher:

$ ./lotus paych voucher best-spendable t242ovcjmx2sfvuxyzzibtoa6xjukola2cqjh7k6a
i1UC5p1RJZfUi1pfGcoDNwPXTRTlg0IAAED2AAFDAAR-AIBYQgFg4MDyo8VGDivv6QmawB9ngXHIISZpVGXDqe2mWennVXusw5A0spST1HEffEsE0Ls_keXddgXf5X3FuurCEDv3AQ
Amount: 1150

If there is some reason why the 1150 voucher is no longer valid, then it should not print anything or print "None" or similar.

Version (run lotus version):

$ lotus version
Daemon:  0.4.1+git.190882ee+api0.8.1
Local: lotus version 0.4.1+git.190882ee

[ghost]

Based on my now much-better understanding of how voucher redemption is supposed to work, I no longer this is a bug. Once you've submitted the best-spendable for a lane, there is nothing more than you can submit.

So, Lotus is right to point out that it can't give an answer to lotus paych voucher best-spendable t2... after the submit. Maybe we could just change the error message to something like "The best spendable voucher has already been submitted."

Also, I now think this issue should be labeled as low priority.

[dirkmc]

That's true, although there's also an issue in that if there are multiple vouchers in a lane, and one of them is submitted, then best-spendable stops working. If for example there are vouchers for 5 and 10 fil, and the one for 5 fil is submitted, then best-spendable should still report the voucher for 10 fil as the best.
This is fixed by #3197

[ghost]

Ah, very nice! I look forward to the merge of #3197.