Better tests for transaction conflict handling #1064

LagginTimes · 2023-08-07T08:19:18Z

Description

This PR introduces a new TxTemplate struct to test different transaction conflict scenarios in TxGraph.
The following transaction conflict scenarios are tested:

2 unconfirmed txs with different last_seens conflict. The most recent tx should be the only tx that appears in the list methods.
3 unconfirmed txs with different last_seens conflict. The most recent tx should be the only tx that appears in the list methods.
An unconfirmed tx U conflicts with a tx anchored in orphaned block O. O has higher last_seen. O should be the only tx that appears in the list methods.
An unconfirmed tx U conflicts with a tx anchored in orphaned block O. U has higher last_seen. U should be the only tx that appears in the list methods.
Multiple unconfirmed txs conflict with a confirmed tx. None of the unconfirmed txs should appear in the list methods.
B and B' conflict. C spends B. B' is anchored in best chain. B and C should not appear in the list methods.
B and B' conflict. C spends B. B is anchored in best chain. B' should not appear in the list methods.
B and B' conflict. C spends both B and B'. C is impossible.
B and B' conflict. C spends both B and B'. C is impossible. B' is confirmed.
B and B' conflict. C spends both B and B'. C is impossible. D spends C.

These tests revealed that TxGraph::walk_conflicts was not checking ancestors of the root tx for conflicts. TxGraph::walk_conflicts has been refactored to check for conflicting ancestor transactions by using a new TxAncestors iterator in TxGraph.

Changelog notice

Introduced tx_template module
Introduced TxGraph::TxAncestors iterator
Refactored TxGraph::walk_conflicts to use TxGraph::TxAncestors
Added walk_ancestors to TxGraph

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

New Features:

I've added tests for the new feature
I've added docs for the new feature

evanlinjin

Thanks for working on this, glad to see the progress. Just some comments!

crates/chain/tests/common/tx_template.rs

crates/chain/tests/test_indexed_tx_graph.rs

crates/chain/tests/common/tx_template.rs

crates/chain/tests/test_indexed_tx_graph.rs

crates/chain/tests/common/tx_template.rs

evanlinjin · 2023-08-17T10:04:36Z

Please rebase on master to fix the CI.

crates/chain/tests/common/tx_template.rs

danielabrozzoni · 2023-09-15T13:28:18Z

I'll ACK this once we fix the commit messages.

In case you have never done that, https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History can be of help :)

evanlinjin

ACK 452627d

LLFourn

Thanks @LagginTimes. I've pointed out what I think is a bug that I think should be fixed here. The other things I've identified I leave to you to resolve them here or we can do it in follow up work.

The key things we should (eventually) resolve are:

The worst case time complexity of the algorithm to determine whether a single tx is in the chain is roughly linear in the number of unconfirmed transactions. If we are determining whether a bunch of txouts are in the chain and every unconfirmed tx has one this means it's quadratic. I fear by peppering the mempool with a bunch of unconfirmed txs and RBFing them a lot you could denial of service attack systems using this algorithm.
I think it'd be better if TxDescendents and TxAncestors should be proximity to the root txid in the graph. The current order is depth first and at what "depth" the node is visited at depends on the order of the outputs or inputs that are being traversed. The iterator can give a depth of 7 to a tx that has a direct relationship to the root tx.

crates/chain/src/tx_graph.rs

danielabrozzoni · 2023-09-29T13:57:29Z

As discussed with @evanlinjin, I'm picking this up as it's high priority :)

Co-authored-by: Wei Chen <wzc110@gmail.com>

This commit also changes test_descendants_no_repeat to check the order of the transactions returned

Co-authored-by: Wei Chen <wzc110@gmail.com>

danielabrozzoni

I just pushed df1ee4f:

Rewrite commit history
Fix review comments

As a part of fixing the review comments I added a scenario in test_tx_graph_conflicts, I just wanted to say @LagginTimes, the Scenario/TxTemplate mechanism is super convenient! Good job :)

crates/chain/src/tx_graph.rs

LLFourn · 2023-10-04T00:11:35Z

crates/chain/tests/test_tx_graph_conflicts.rs

+ },
+ ],
+ // B and C should not appear in the list methods
+ exp_chain_txs: HashSet::from(["A", "B'"]),


Isn't what we want to happen here is that B and C are canonical?

C has the more recent last seen which should transitively mean that B is chosen over B' . In other words the effective last_seen of a node is the max of all its descendants.

This PR at least makes the answer consistent but it seems we are actually asserting suboptimal behavior here. Can be done in a follow up.

Just to give a bit of context, my reasoning behind this was: how would a node react if it saw B, then B', then C? And I think it would go like:

B: fine, I'll keep this

B': fine, I'll keep this, and drop B

C: I'll drop this, as it's spending B, which I dropped already

Which I don't know if it's coherent with the rest of the txgraph and the last_seen logic (it probably isn't).

Anyways, I just pushed a fix, it should be fine now :)

Which I don't know if it's coherent with the rest of the txgraph and the last_seen logic (it probably isn't).

Indeed. TxGraph is montone so doesn't record the order of things entering into it. Our conflict resolution mechanism is last_seen so I think we should take whichever subgraph has the tx with the latest one. Hopefully we can make this efficient!

LLFourn

Approved. Consider fixing https://github.com/bitcoindevkit/bdk/pull/1064/files#r1344900282
ACK df1ee4f

...try_get_chain_pos In try_get_chain_pos, when we notice that a transaction is not included in the best chain, we check the transactions in mempool to find conflicting ones, and decide based on that if our transaction is still in mempool or has been dropped. This commit adds a check for transactions conflicting with the unconfirmed ancestors of our tx. Co-authored-by: Wei Chen <wzc110@gmail.com>

Co-authored-by: Wei Chen <wzc110@gmail.com>

evanlinjin

ACK 6d601a7

Thanks @danielabrozzoni for taking this forward and @LagginTimes for the initial work.

afbf83c chain(fix): conflict resolution for txs with same last_seen (Wei Chen) Pull request description: ### Description Fixes #1102. If a conflicting tx has the same `last_seen`, then we check lexicographical sorting of txids. ### Notes to the reviewers The tests for this fix exist in the `TxTemplate` structure in #1064 which may need to be merged first. ### Checklists #### All Submissions: * [x] I've signed all my commits * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md) * [x] I ran `cargo fmt` and `cargo clippy` before committing ACKs for top commit: danielabrozzoni: ACK afbf83c Tree-SHA512: 91b8fbff305b715247501b861ab7ea9e9d9ef99248b05d14e01aacf7e64ad7826f35773e8998cf421dbd04f663714026084c6e817ac5365bce4844c8ea3b7e3f

LagginTimes marked this pull request as draft August 7, 2023 08:23

LagginTimes force-pushed the tx_conflict_tests branch from 952930b to cd45f15 Compare August 9, 2023 18:21

notmandatory assigned LagginTimes Aug 9, 2023

notmandatory added the tests label Aug 9, 2023

LagginTimes force-pushed the tx_conflict_tests branch from cd45f15 to 4d7c3fb Compare August 10, 2023 10:44

evanlinjin reviewed Aug 11, 2023

View reviewed changes

crates/chain/tests/common/tx_template.rs Outdated Show resolved Hide resolved

crates/chain/tests/test_indexed_tx_graph.rs Outdated Show resolved Hide resolved

crates/chain/tests/test_indexed_tx_graph.rs Outdated Show resolved Hide resolved

LagginTimes force-pushed the tx_conflict_tests branch 2 times, most recently from d112807 to 83c92b6 Compare August 11, 2023 10:06

evanlinjin reviewed Aug 11, 2023

View reviewed changes

crates/chain/tests/test_indexed_tx_graph.rs Outdated Show resolved Hide resolved

LagginTimes force-pushed the tx_conflict_tests branch 3 times, most recently from ce6cb40 to 67d9a5e Compare August 14, 2023 10:59

evanlinjin reviewed Aug 14, 2023

View reviewed changes

crates/chain/tests/test_indexed_tx_graph.rs Outdated Show resolved Hide resolved

evanlinjin reviewed Aug 15, 2023

View reviewed changes

crates/chain/tests/common/tx_template.rs Outdated Show resolved Hide resolved

LagginTimes force-pushed the tx_conflict_tests branch from 67d9a5e to 70d010c Compare August 15, 2023 09:28

evanlinjin reviewed Aug 15, 2023

View reviewed changes

crates/chain/tests/common/tx_template.rs Outdated Show resolved Hide resolved

LagginTimes force-pushed the tx_conflict_tests branch from 70d010c to 4e1d898 Compare August 15, 2023 11:41

nondiremanuel mentioned this pull request Aug 15, 2023

W33 Lib Team Call bitcoindevkit/.github#72

Closed

26 tasks

nondiremanuel added this to the 1.0.0-alpha.2 milestone Aug 15, 2023

LagginTimes force-pushed the tx_conflict_tests branch from 4e1d898 to a86f76a Compare August 15, 2023 13:23

evanlinjin reviewed Aug 16, 2023

View reviewed changes

crates/chain/tests/test_indexed_tx_graph.rs Outdated Show resolved Hide resolved

LagginTimes force-pushed the tx_conflict_tests branch 3 times, most recently from af19b5a to 238f0f1 Compare August 17, 2023 09:50

evanlinjin reviewed Aug 17, 2023

View reviewed changes

crates/chain/tests/common/tx_template.rs Outdated Show resolved Hide resolved

evanlinjin reviewed Aug 17, 2023

View reviewed changes

crates/chain/tests/common/tx_template.rs Outdated Show resolved Hide resolved

LagginTimes force-pushed the tx_conflict_tests branch 2 times, most recently from 112e851 to b00b406 Compare August 22, 2023 09:27

nondiremanuel mentioned this pull request Aug 22, 2023

W35 Lib Team Call bitcoindevkit/.github#74

Closed

30 tasks

LagginTimes force-pushed the tx_conflict_tests branch 3 times, most recently from 7340a41 to 452627d Compare September 15, 2023 14:31

evanlinjin approved these changes Sep 15, 2023

View reviewed changes

LLFourn requested changes Sep 19, 2023

View reviewed changes

nondiremanuel mentioned this pull request Sep 23, 2023

W39 Lib Team Call bitcoindevkit/.github#78

Closed

40 tasks

danielabrozzoni mentioned this pull request Sep 29, 2023

TxGraph::direct_conflicts_of_tx should just be called direct_conflicts #1138

Closed

evanlinjin mentioned this pull request Oct 3, 2023

Coinbase transaction that is not anchored in the best chain should always be filtered out #1144

Closed

danielabrozzoni and others added 4 commits October 3, 2023 11:25

doc(chain): Fix typos

486e0e1

Co-authored-by: Wei Chen <wzc110@gmail.com>

fix(chain): TxDescendants performs a BFS

2f26eca

This commit also changes test_descendants_no_repeat to check the order of the transactions returned

feat(chain): Introduce TxAncestors, walk_ancestors

4742d88

Co-authored-by: Wei Chen <wzc110@gmail.com>

doc(chain): Clarify direct_conflicts_of_tx's docs

a3e8480

Co-authored-by: Wei Chen <wzc110@gmail.com>

danielabrozzoni force-pushed the tx_conflict_tests branch from 452627d to df1ee4f Compare October 3, 2023 13:18

danielabrozzoni reviewed Oct 3, 2023

View reviewed changes

crates/chain/src/tx_graph.rs Outdated Show resolved Hide resolved

crates/chain/src/tx_graph.rs Show resolved Hide resolved

crates/chain/src/tx_graph.rs Outdated Show resolved Hide resolved

LLFourn reviewed Oct 4, 2023

View reviewed changes

LLFourn approved these changes Oct 4, 2023

View reviewed changes

danielabrozzoni and others added 5 commits October 4, 2023 14:00

test(chain): add block_id! utility macro

6e51147

Co-authored-by: Wei Chen <wzc110@gmail.com>

test(chain): Introduce TxTemplate

59a2403

Co-authored-by: Wei Chen <wzc110@gmail.com>

test(chain): Add test for walk_ancestors

48ca95b

Co-authored-by: Wei Chen <wzc110@gmail.com>

test(chain): Add test for conflicting transactions

6d601a7

Co-authored-by: Wei Chen <wzc110@gmail.com>

danielabrozzoni force-pushed the tx_conflict_tests branch from df1ee4f to 6d601a7 Compare October 4, 2023 12:02

evanlinjin approved these changes Oct 5, 2023

View reviewed changes

evanlinjin merged commit 4ee11aa into bitcoindevkit:master Oct 5, 2023
12 checks passed

This was referenced Oct 6, 2023

More tests for IndexedTxGraph and TxGraph #958

Closed

W41 Lib Team Call bitcoindevkit/.github#80

Closed

notmandatory mentioned this pull request Oct 11, 2023

Release 1.0.0-alpha.2 #1168

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better tests for transaction conflict handling #1064

Better tests for transaction conflict handling #1064

LagginTimes commented Aug 7, 2023 •

edited

Loading

evanlinjin left a comment

evanlinjin commented Aug 17, 2023

danielabrozzoni commented Sep 15, 2023

evanlinjin left a comment

LLFourn left a comment

danielabrozzoni commented Sep 29, 2023

danielabrozzoni left a comment

LLFourn Oct 4, 2023

danielabrozzoni Oct 4, 2023

LLFourn Oct 6, 2023

LLFourn left a comment

evanlinjin left a comment

Better tests for transaction conflict handling #1064

Better tests for transaction conflict handling #1064

Conversation

LagginTimes commented Aug 7, 2023 • edited Loading

Description

Changelog notice

Checklists

New Features:

evanlinjin left a comment

Choose a reason for hiding this comment

evanlinjin commented Aug 17, 2023

danielabrozzoni commented Sep 15, 2023

evanlinjin left a comment

Choose a reason for hiding this comment

LLFourn left a comment

Choose a reason for hiding this comment

danielabrozzoni commented Sep 29, 2023

danielabrozzoni left a comment

Choose a reason for hiding this comment

LLFourn Oct 4, 2023

Choose a reason for hiding this comment

danielabrozzoni Oct 4, 2023

Choose a reason for hiding this comment

LLFourn Oct 6, 2023

Choose a reason for hiding this comment

LLFourn left a comment

Choose a reason for hiding this comment

evanlinjin left a comment

Choose a reason for hiding this comment

LagginTimes commented Aug 7, 2023 •

edited

Loading