Skip to content

Commit 7f647bb

Browse files
Merge #6517: backport: merge bitcoin#25497, bitcoin#17331, bitcoin#22155 (wallet backports: part 1)
153bdc2 merge bitcoin#22155: Add test for subtract fee from recipient behavior (Kittywhiskers Van Gogh) 0185847 fix: correct fee calculations in `CreateTransactionInternal` (Kittywhiskers Van Gogh) 445f489 merge bitcoin#17331: Use effective values throughout coin selection (Kittywhiskers Van Gogh) 7e54bd9 wallet: copy and sort `vecSend` if BIP69 sorting enabled, enable sorting (Kittywhiskers Van Gogh) 9e9e66f partial bitcoin#17331: Use effective values throughout coin selection (Kittywhiskers Van Gogh) 66fe2d4 merge bitcoin#25497: more accurate target for large transactions (Kittywhiskers Van Gogh) 6e4d789 wallet: add back missing `CoinSelectionParams` assignments (Kittywhiskers Van Gogh) bd35042 wallet: move `CoinSelectionParams` to positions expected upstream (Kittywhiskers Van Gogh) 0711e67 wallet: shuffle transaction inputs if we aren't using BIP69 (Kittywhiskers Van Gogh) 1cf1c58 refactor: move selected coin and txin sorting to the end of the scope (Kittywhiskers Van Gogh) ab756ba wallet: Fail if maximum weight is too large (Kittywhiskers Van Gogh) 05c319e refactor: move oversized transaction check to tail end of scope (Kittywhiskers Van Gogh) 6ca51df wallet: Use existing feerate instead of getting a new one (Kittywhiskers Van Gogh) Pull request description: ## Additional Information * Dependent on #6543 * Dependency for #6529 * [bitcoin#17331](bitcoin#17331) logically partially reverts [dash#3368](#3668) as Dash Core implemented a calculate-before approach (compared to _then_ Bitcoin Core's calculate-and-adjust approach) and it is being replaced with the current upstream calculate-after approach done in a single-pass instead of iteratively (like the former two). * As the changes are non-trivial, they have been split into a "partial" and a "merge" commit, the first half dedicated just to the logical partial revert and the latter half dedicated to using effective values in coin selection. * BIP69 sorting is disabled in the former half to allow the fix to be in a separate commit while allowing validation of the remaining set of changes. The fix re-enables BIP69 sorting. * Due to the changes introduced in [dash#3368](#3668), a lot of then-redundant code was removed and changes to it upstream were not mirrored in Dash Core. To allow [bitcoin#17331](bitcoin#17331) to work properly, a lot of that unmirrored code was reintroduced and existing code readjusted to match upstream. * `coin_selection_params.tx_noinputs_size` is said to have a size (sans output count) of `9` instead of `10` as we don't have a SegWit field (referred to as `1 witness overhead (dummy, flag, stack size)` in a code comment) on account of not having SegWit. * To allow for backporting [bitcoin#17331](bitcoin#17331), portions of [bitcoin#21083](bitcoin#21083) (1a6a0b0) and [bitcoin#20536](bitcoin@51e2cd3) (3e69939) were backported. * [bitcoin#17331](bitcoin#17331) seems to have silently broken `CreateTransactionInternal` as functional tests fail (see below) despite the backport not intending to change behavior. This was caught due to unit tests introduced in [dash#3667](#3667). The aberration seems be remedied by portions of [bitcoin#25647](bitcoin#25647) and [bitcoin#26643](bitcoin#26643) and they have been incorporated into this pull request in a separate commit. **Special thanks to UdjinM6 for figuring this out!** 🎉 <details> <summary>Error log:</summary> ``` dash@479e0aa4ebbf:/src/dash$ ./src/test/test_dash -t coinselector_tests,wallet_tests Running 21 test cases... wallet/test/wallet_tests.cpp(749): error: in "wallet_tests/CreateTransactionTest": check expected == actual has failed [false != true] CreateTransactionTest failed at: 2 - 5 wallet/test/wallet_tests.cpp(749): error: in "wallet_tests/CreateTransactionTest": check expected == actual has failed [false != true] CreateTransactionTest failed at: 4 - 4 wallet/test/wallet_tests.cpp(749): error: in "wallet_tests/CreateTransactionTest": check expected == actual has failed [false != true] CreateTransactionTest failed at: 4 - 5 wallet/test/wallet_tests.cpp(749): error: in "wallet_tests/CreateTransactionTest": check expected == actual has failed [false != true] CreateTransactionTest failed at: 6 - 0 wallet/test/wallet_tests.cpp(749): error: in "wallet_tests/CreateTransactionTest": check expected == actual has failed [false != true] CreateTransactionTest failed at: 6 - 2 wallet/test/wallet_tests.cpp(749): error: in "wallet_tests/CreateTransactionTest": check expected == actual has failed [false != true] CreateTransactionTest failed at: 6 - 4 wallet/test/wallet_tests.cpp(749): error: in "wallet_tests/CreateTransactionTest": check expected == actual has failed [false != true] CreateTransactionTest failed at: 6 - 5 *** 7 failures are detected in the test module "Dash Core Test Suite" ``` </details> ## How Has This Been Tested? 153bdc2 was tested on Debian 12 (`bookworm`) mixing ~2 tDASH on default settings. ![CoinJoin run](https://github.com/user-attachments/assets/da1f13e7-dd83-4211-8d42-0cd4c770bbf1) ## Breaking Changes * If a transaction isn't shuffled using BIP69 (i.e. if an explicit position for the change txout is specified), it will be randomly shuffled (mirroring upstream behavior, [source](https://github.com/bitcoin/bitcoin/blob/51a3ac242c92e69b59df26f8f9e287b31e5c3b0f/src/wallet/wallet.cpp#L3048)). This deviates from earlier behavior where no shuffling would be done at all if BIP69 isn't applied. ## Checklist - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have added or updated relevant unit/integration/functional/e2e tests - [x] I have made corresponding changes to the documentation **(note: N/A)** - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ ACKs for top commit: UdjinM6: utACK 153bdc2 PastaPastaPasta: utACK 153bdc2 Tree-SHA512: 709b77dce3cea2bbf09eab42c7e70171c3bc6527ff4c387a4db75994da5c0d59025b641d90f734b87a62cdfa8e422d09513697a6e875635de2765a1c9141233e
2 parents 078d6ea + 153bdc2 commit 7f647bb

File tree

10 files changed

+445
-453
lines changed

10 files changed

+445
-453
lines changed

src/Makefile.test.include

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,7 @@ BITCOIN_TESTS += \
194194
wallet/test/bip39_tests.cpp \
195195
wallet/test/coinjoin_tests.cpp \
196196
wallet/test/psbt_wallet_tests.cpp \
197+
wallet/test/spend_tests.cpp \
197198
wallet/test/wallet_tests.cpp \
198199
wallet/test/walletdb_tests.cpp \
199200
wallet/test/wallet_crypto_tests.cpp \

src/bench/coin_selection.cpp

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -48,15 +48,14 @@ static void CoinSelection(benchmark::Bench& bench)
4848
coins.emplace_back(wtx.get(), 0 /* iIn */, 6 * 24 /* nDepthIn */, true /* spendable */, true /* solvable */, true /* safe */);
4949
}
5050
const CoinEligibilityFilter filter_standard(1, 6, 0);
51-
const CoinSelectionParams coin_selection_params(/* use_bnb= */ true, /* change_output_size= */ 34,
51+
const CoinSelectionParams coin_selection_params(/* change_output_size= */ 34,
5252
/* change_spend_size= */ 148, /* effective_feerate= */ CFeeRate(0),
5353
/* long_term_feerate= */ CFeeRate(0), /* discard_feerate= */ CFeeRate(0),
5454
/* tx_no_inputs_size= */ 0, /* avoid_partial= */ false);
5555
bench.run([&] {
5656
std::set<CInputCoin> setCoinsRet;
5757
CAmount nValueRet;
58-
bool bnb_used;
59-
bool success = wallet.SelectCoinsMinConf(1003 * COIN, filter_standard, coins, setCoinsRet, nValueRet, coin_selection_params, bnb_used);
58+
bool success = wallet.SelectCoinsMinConf(1003 * COIN, filter_standard, coins, setCoinsRet, nValueRet, coin_selection_params);
6059
assert(success);
6160
assert(nValueRet == 1003 * COIN);
6261
assert(setCoinsRet.size() == 2);
@@ -94,12 +93,11 @@ static void BnBExhaustion(benchmark::Bench& bench)
9493
std::vector<OutputGroup> utxo_pool;
9594
CoinSet selection;
9695
CAmount value_ret = 0;
97-
CAmount not_input_fees = 0;
9896

9997
bench.run([&] {
10098
// Benchmark
10199
CAmount target = make_hard_case(17, utxo_pool);
102-
SelectCoinsBnB(utxo_pool, target, 0, selection, value_ret, not_input_fees); // Should exhaust
100+
SelectCoinsBnB(utxo_pool, target, 0, selection, value_ret); // Should exhaust
103101

104102
// Cleanup
105103
utxo_pool.clear();

src/wallet/coinselection.cpp

Lines changed: 27 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
struct {
1717
bool operator()(const OutputGroup& a, const OutputGroup& b) const
1818
{
19-
return a.effective_value > b.effective_value;
19+
return a.GetSelectionAmount() > b.GetSelectionAmount();
2020
}
2121
} descending;
2222

@@ -51,37 +51,34 @@ struct {
5151
* @param const std::vector<CInputCoin>& utxo_pool The set of UTXOs that we are choosing from.
5252
* These UTXOs will be sorted in descending order by effective value and the CInputCoins'
5353
* values are their effective values.
54-
* @param const CAmount& target_value This is the value that we want to select. It is the lower
54+
* @param const CAmount& selection_target This is the value that we want to select. It is the lower
5555
* bound of the range.
5656
* @param const CAmount& cost_of_change This is the cost of creating and spending a change output.
57-
* This plus target_value is the upper bound of the range.
57+
* This plus selection_target is the upper bound of the range.
5858
* @param std::set<CInputCoin>& out_set -> This is an output parameter for the set of CInputCoins
5959
* that have been selected.
6060
* @param CAmount& value_ret -> This is an output parameter for the total value of the CInputCoins
6161
* that were selected.
62-
* @param CAmount not_input_fees -> The fees that need to be paid for the outputs and fixed size
63-
* overhead (version, locktime, marker and flag)
6462
*/
6563

6664
static const size_t TOTAL_TRIES = 100000;
6765

68-
bool SelectCoinsBnB(std::vector<OutputGroup>& utxo_pool, const CAmount& target_value, const CAmount& cost_of_change, std::set<CInputCoin>& out_set, CAmount& value_ret, CAmount not_input_fees)
66+
bool SelectCoinsBnB(std::vector<OutputGroup>& utxo_pool, const CAmount& selection_target, const CAmount& cost_of_change, std::set<CInputCoin>& out_set, CAmount& value_ret)
6967
{
7068
out_set.clear();
7169
CAmount curr_value = 0;
7270

7371
std::vector<bool> curr_selection; // select the utxo at this index
7472
curr_selection.reserve(utxo_pool.size());
75-
CAmount actual_target = not_input_fees + target_value;
7673

7774
// Calculate curr_available_value
7875
CAmount curr_available_value = 0;
7976
for (const OutputGroup& utxo : utxo_pool) {
8077
// Assert that this utxo is not negative. It should never be negative, effective value calculation should have removed it
81-
assert(utxo.effective_value > 0);
82-
curr_available_value += utxo.effective_value;
78+
assert(utxo.GetSelectionAmount() > 0);
79+
curr_available_value += utxo.GetSelectionAmount();
8380
}
84-
if (curr_available_value < actual_target) {
81+
if (curr_available_value < selection_target) {
8582
return false;
8683
}
8784

@@ -96,12 +93,12 @@ bool SelectCoinsBnB(std::vector<OutputGroup>& utxo_pool, const CAmount& target_v
9693
for (size_t i = 0; i < TOTAL_TRIES; ++i) {
9794
// Conditions for starting a backtrack
9895
bool backtrack = false;
99-
if (curr_value + curr_available_value < actual_target || // Cannot possibly reach target with the amount remaining in the curr_available_value.
100-
curr_value > actual_target + cost_of_change || // Selected value is out of range, go back and try other branch
96+
if (curr_value + curr_available_value < selection_target || // Cannot possibly reach target with the amount remaining in the curr_available_value.
97+
curr_value > selection_target + cost_of_change || // Selected value is out of range, go back and try other branch
10198
(curr_waste > best_waste && (utxo_pool.at(0).fee - utxo_pool.at(0).long_term_fee) > 0)) { // Don't select things which we know will be more wasteful if the waste is increasing
10299
backtrack = true;
103-
} else if (curr_value >= actual_target) { // Selected value is within range
104-
curr_waste += (curr_value - actual_target); // This is the excess value which is added to the waste for the below comparison
100+
} else if (curr_value >= selection_target) { // Selected value is within range
101+
curr_waste += (curr_value - selection_target); // This is the excess value which is added to the waste for the below comparison
105102
// Adding another UTXO after this check could bring the waste down if the long term fee is higher than the current fee.
106103
// However we are not going to explore that because this optimization for the waste is only done when we have hit our target
107104
// value. Adding any more UTXOs will be just burning the UTXO; it will go entirely to fees. Thus we aren't going to
@@ -114,7 +111,7 @@ bool SelectCoinsBnB(std::vector<OutputGroup>& utxo_pool, const CAmount& target_v
114111
break;
115112
}
116113
}
117-
curr_waste -= (curr_value - actual_target); // Remove the excess value as we will be selecting different coins now
114+
curr_waste -= (curr_value - selection_target); // Remove the excess value as we will be selecting different coins now
118115
backtrack = true;
119116
}
120117

@@ -123,7 +120,7 @@ bool SelectCoinsBnB(std::vector<OutputGroup>& utxo_pool, const CAmount& target_v
123120
// Walk backwards to find the last included UTXO that still needs to have its omission branch traversed.
124121
while (!curr_selection.empty() && !curr_selection.back()) {
125122
curr_selection.pop_back();
126-
curr_available_value += utxo_pool.at(curr_selection.size()).effective_value;
123+
curr_available_value += utxo_pool.at(curr_selection.size()).GetSelectionAmount();
127124
}
128125

129126
if (curr_selection.empty()) { // We have walked back to the first utxo and no branch is untraversed. All solutions searched
@@ -133,24 +130,24 @@ bool SelectCoinsBnB(std::vector<OutputGroup>& utxo_pool, const CAmount& target_v
133130
// Output was included on previous iterations, try excluding now.
134131
curr_selection.back() = false;
135132
OutputGroup& utxo = utxo_pool.at(curr_selection.size() - 1);
136-
curr_value -= utxo.effective_value;
133+
curr_value -= utxo.GetSelectionAmount();
137134
curr_waste -= utxo.fee - utxo.long_term_fee;
138135
} else { // Moving forwards, continuing down this branch
139136
OutputGroup& utxo = utxo_pool.at(curr_selection.size());
140137

141138
// Remove this utxo from the curr_available_value utxo amount
142-
curr_available_value -= utxo.effective_value;
139+
curr_available_value -= utxo.GetSelectionAmount();
143140

144141
// Avoid searching a branch if the previous UTXO has the same value and same waste and was excluded. Since the ratio of fee to
145142
// long term fee is the same, we only need to check if one of those values match in order to know that the waste is the same.
146143
if (!curr_selection.empty() && !curr_selection.back() &&
147-
utxo.effective_value == utxo_pool.at(curr_selection.size() - 1).effective_value &&
144+
utxo.GetSelectionAmount() == utxo_pool.at(curr_selection.size() - 1).GetSelectionAmount() &&
148145
utxo.fee == utxo_pool.at(curr_selection.size() - 1).fee) {
149146
curr_selection.push_back(false);
150147
} else {
151148
// Inclusion branch first (Largest First Exploration)
152149
curr_selection.push_back(true);
153-
curr_value += utxo.effective_value;
150+
curr_value += utxo.GetSelectionAmount();
154151
curr_waste += utxo.fee - utxo.long_term_fee;
155152
}
156153
}
@@ -283,14 +280,14 @@ bool KnapsackSolver(const CAmount& nTargetValue, std::vector<OutputGroup>& group
283280
if (tryDenom == 0 && CoinJoin::IsDenominatedAmount(group.m_value)) {
284281
continue; // we don't want denom values on first run
285282
}
286-
if (group.m_value == nTargetValue) {
283+
if (group.GetSelectionAmount() == nTargetValue) {
287284
util::insert(setCoinsRet, group.m_outputs);
288285
nValueRet += group.m_value;
289286
return true;
290-
} else if (group.m_value < nTargetValue + nMinChange) {
287+
} else if (group.GetSelectionAmount() < nTargetValue + nMinChange) {
291288
applicable_groups.push_back(group);
292-
nTotalLower += group.m_value;
293-
} else if (!lowest_larger || group.m_value < lowest_larger->m_value) {
289+
nTotalLower += group.GetSelectionAmount();
290+
} else if (!lowest_larger || group.GetSelectionAmount() < lowest_larger->GetSelectionAmount()) {
294291
lowest_larger = group;
295292
}
296293
}
@@ -336,7 +333,7 @@ bool KnapsackSolver(const CAmount& nTargetValue, std::vector<OutputGroup>& group
336333
// If we have a bigger coin and (either the stochastic approximation didn't find a good solution,
337334
// or the next bigger coin is closer), return the bigger coin
338335
if (lowest_larger &&
339-
((nBest != nTargetValue && nBest < nTargetValue + nMinChange) || lowest_larger->m_value <= nBest)) {
336+
((nBest != nTargetValue && nBest < nTargetValue + nMinChange) || lowest_larger->GetSelectionAmount() <= nBest)) {
340337
util::insert(setCoinsRet, lowest_larger->m_outputs);
341338
nValueRet += lowest_larger->m_value;
342339
} else {
@@ -400,3 +397,8 @@ bool OutputGroup::EligibleForSpending(const CoinEligibilityFilter& eligibility_f
400397
&& m_ancestors <= eligibility_filter.max_ancestors
401398
&& m_descendants <= eligibility_filter.max_descendants;
402399
}
400+
401+
CAmount OutputGroup::GetSelectionAmount() const
402+
{
403+
return m_subtract_fee_outputs ? m_value : effective_value;
404+
}

src/wallet/coinselection.h

Lines changed: 48 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,45 @@ class CInputCoin {
5757
}
5858
};
5959

60+
/** Parameters for one iteration of Coin Selection. */
61+
struct CoinSelectionParams
62+
{
63+
/** Size of a change output in bytes, determined by the output type. */
64+
size_t change_output_size = 0;
65+
/** Size of the input to spend a change output in virtual bytes. */
66+
size_t change_spend_size = 0;
67+
/** Cost of creating the change output. */
68+
CAmount m_change_fee{0};
69+
/** Cost of creating the change output + cost of spending the change output in the future. */
70+
CAmount m_cost_of_change{0};
71+
/** The fee to spend these UTXOs at the long term feerate. */
72+
CFeeRate m_effective_feerate;
73+
/** The feerate estimate used to estimate an upper bound on what should be sufficient to spend
74+
* the change output sometime in the future. */
75+
CFeeRate m_long_term_feerate;
76+
/** If the cost to spend a change output at the discard feerate exceeds its value, drop it to fees. */
77+
CFeeRate m_discard_feerate;
78+
size_t tx_noinputs_size = 0;
79+
/** Indicate that we are subtracting the fee from outputs */
80+
bool m_subtract_fee_outputs = false;
81+
/** When true, always spend all (up to OUTPUT_GROUP_MAX_ENTRIES) or none of the outputs
82+
* associated with the same address. This helps reduce privacy leaks resulting from address
83+
* reuse. Dust outputs are not eligible to be added to output groups and thus not considered. */
84+
bool m_avoid_partial_spends = false;
85+
86+
CoinSelectionParams(size_t change_output_size, size_t change_spend_size, CFeeRate effective_feerate,
87+
CFeeRate long_term_feerate, CFeeRate discard_feerate, size_t tx_noinputs_size, bool avoid_partial) :
88+
change_output_size(change_output_size),
89+
change_spend_size(change_spend_size),
90+
m_effective_feerate(effective_feerate),
91+
m_long_term_feerate(long_term_feerate),
92+
m_discard_feerate(discard_feerate),
93+
tx_noinputs_size(tx_noinputs_size),
94+
m_avoid_partial_spends(avoid_partial)
95+
{}
96+
CoinSelectionParams() {}
97+
};
98+
6099
/** Parameters for filtering which OutputGroups we may use in coin selection.
61100
* We start by being very selective and requiring multiple confirmations and
62101
* then get more permissive if we cannot fund the transaction. */
@@ -109,18 +148,23 @@ struct OutputGroup
109148
* a lower feerate). Calculated using long term fee estimate. This is used to decide whether
110149
* it could be economical to create a change output. */
111150
CFeeRate m_long_term_feerate{0};
151+
/** Indicate that we are subtracting the fee from outputs.
152+
* When true, the value that is used for coin selection is the UTXO's real value rather than effective value */
153+
bool m_subtract_fee_outputs{false};
112154

113155
OutputGroup() {}
114-
OutputGroup(const CFeeRate& effective_feerate, const CFeeRate& long_term_feerate) :
115-
m_effective_feerate(effective_feerate),
116-
m_long_term_feerate(long_term_feerate)
156+
OutputGroup(const CoinSelectionParams& params) :
157+
m_effective_feerate(params.m_effective_feerate),
158+
m_long_term_feerate(params.m_long_term_feerate),
159+
m_subtract_fee_outputs(params.m_subtract_fee_outputs)
117160
{}
118161

119162
void Insert(const CInputCoin& output, int depth, bool from_me, size_t ancestors, size_t descendants, bool positive_only);
120163
bool EligibleForSpending(const CoinEligibilityFilter& eligibility_filter, bool isISLocked) const;
164+
CAmount GetSelectionAmount() const;
121165
};
122166

123-
bool SelectCoinsBnB(std::vector<OutputGroup>& utxo_pool, const CAmount& target_value, const CAmount& cost_of_change, std::set<CInputCoin>& out_set, CAmount& value_ret, CAmount not_input_fees);
167+
bool SelectCoinsBnB(std::vector<OutputGroup>& utxo_pool, const CAmount& selection_target, const CAmount& cost_of_change, std::set<CInputCoin>& out_set, CAmount& value_ret);
124168

125169
// Original coin selection algorithm as a fallback
126170
bool KnapsackSolver(const CAmount& nTargetValue, std::vector<OutputGroup>& groups, std::set<CInputCoin>& setCoinsRet, CAmount& nValueRet, bool fFulyMixedOnly, CAmount maxTxFee);

0 commit comments

Comments
 (0)