Absolute fee constraint in TxBuilder not being respected

[sjeohp]

When creating a tx the absolute fee amount when set does not always match the actual fee amount. I see this is because the amount is added to several times, so it must be deliberate, but for someone using the library it is unintuitive. What is the rationale?

[notmandatory]

Can you also give a little code example and which version of bdk are you using? This would help us know which documentation needs to be improved.

[sjeohp]

crates/bdk/wallet/mod.rs
fn create_tx

[sjeohp]

For example

let mut tx_builder = wallet.build_tx();
tx_builder.fee_absolute(fee);
let (psbt, details) = tx_builder.finish()?;

In my experience the fee will not necessarily match the fee that was set.

[LLFourn]

It's worth pointing out that the "absolute fee" is a really a minimum absolute fee -- it's possible to overshoot it slightly since adding a change output to drian the remaining excess might not be viable.

How far off is the fee vs expected?

[sjeohp]

Making tiny txs on testnet the fee varies by 100-150%.

Is there a way to upper bound it? Or reduce variability at the cost of something else?

[LLFourn]

Appreciated the work looking into this. No but I am not convinced that the implementation isn't simply wrong with respect to absolute fees. Is the fee wrong even when a change output is added? That would be the next question since the builder adds a change output and the resulting tx does not have the correct fee that would mean there's a bug I think.

[sjeohp]

Yes same thing with the change descriptor.

With <desc> having 4500 satoshis and <change_desc> having 1000 satoshis or 0 (no difference):
bdk-cli wallet -d <desc> -c <change_desc> create_tx --to <receiver_addr>:500 --fee_abs 250 returns a fee of 500 most times, sometimes 250.

Also I don't exactly understand this:

bdk-cli wallet -d <DESC_A> -c <DESC_B> get_balance returns the balance of both wallets combined but needs to be synced with both combined. If both are synced separately or inverted like bdk-cli wallet -d <DESC_B> -c <DESC_A> sync the balance will be wrong and create_tx might fail.

[LLFourn]

Yes same thing with the change descriptor.

With <desc> having 4500 satoshis and <change_desc> having 1000 satoshis or 0 (no difference): bdk-cli wallet -d <desc> -c <change_desc> create_tx --to <receiver_addr>:500 --fee_abs 250 returns a fee of 500 most times, sometimes 250.

This does sound like a bug to me. Can you just check that we can distinguish the cases via whether change output is added vs when it is not. i.e. is it always 500 when a change output is added but when you have an output that is close enough to the target amount (here 500) it goes to 250 if not change output is added. In other words i think the bug occurs in the path that change output is added. The culprit is likely a change in or around #641. Strange that tests don't pick it up.

[sjeohp]

Same behaviour without the change flag.

[LLFourn]

Same behaviour without the change flag.

Sorry I don't mean the flag. BDK will add a change output regardless of whether you add a change descriptor (it will just use the main descriptor for the change script pubkey). The question is about whether the change output correlates being added correlates with the bug. You'd need to check the number of outputs the tx has to confirm/deny this. If there's two then a change was added but if there's only one then it was not.

[sjeohp]

Yes you are right the 250 fee corresponds with two outputs, the 500 with only one.

[sjeohp]

What might cause the random difference in number of outputs? Fee estimate? or coin selection algorithm?

[sjeohp]

The issue is when there is no exact match utxos the single_random_draw can result in utxo selection with a remainder that is below the dust threshold, in which case it is added to the fee. IMO this results in unintuitive behaviour in the situations above but is not exactly a bug since whether it is best to create a change output for tiny remainders or add it to the fee is not clear. (It can be cheaper in the long term to not create the change output). Probably if fee_abs is set and exceeded the operation should fail. At any rate I believe the fee should only fluctuate by amounts comparable to the dust threshold.

[LLFourn]

Right this is what I was suggesting here: #1066 (comment)

I doubted it though because it seems unlikely that it would result in exactly 250 error (such a round number). Why would not adding a change output lead to an excess fee of exactly double the requirement -- I guess it's just because you were using rather round numbers for inputs and outputs in the experiment...

[danielabrozzoni]

I've opened #1158 and #1159 to clarify the meaning of absolute_fee