[Taproot API Project] replace `TaprootSpendInfo` with new miniscript-specific structure #815

apoelstra · 2025-05-02T16:50:34Z

In Miniscript, to compute control blocks, estimate satisfaction costs, or otherwise iterate through all the leaves of a Taptree, we use the bitcoin::TaprootSpendInfo structure to maintain a map of all leaves. This map is inappropriate for Miniscript (it may not be appropriate for anyone actually..) for a few reasons:

It is a map from Tapleaves' encoding as Script to data about the Tapleaves; but in Miniscript the Script encoding isn't primary and isn't even available for non-ToPublicKey keys
This map structure means that if duplicate leaves exist then only one of the dupes will be accessible.
The map structure is also really inefficient; it stores the entire merkle path for each leaf even those these paths significantly overlap, leading to O(n log n) space instead of O(n).
Furthermore, we don't need any map because we only ever iterate through the entire tree.

We fix all these issues by introducing a new TrSpendInfo struct which stores the entire Merkle tree in a flat representation and can produce an iterator over all the leaves. The iterator item can be used to access the Script, the Miniscript, the leaf version, and the control block for each leaf, while the TrSpendInfo structure itself can be used to access the internal and external keys. In other words, this one structure efficiently implements APIs for everything that rust-miniscript needs.

This completes the Taproot API overhaul project. After this I'll go back to fixing error types, eliminating recursive structures, or overhauling the validation parameters, whichever one seems most tractable from the current state of master.

We have the function `update_item_with_descriptor_helper` which does a few things: it derives a descriptor (replacing all the xpubs with actual public keys) and updates the appropriate input or output map to map the derived keys to their keysources. It treats Tr outputs differently from other kinds of outputs, because the relevant maps are different. However, in doing so, it duplicates a bunch of work in ways that are hard to follow. Essentially, the algorithm does three things: (a) derives all the keys (and the descriptor), (b) optionally checks that the resulting scriptpubkey is what we expect, and (c) updates the maps. The existing code handles (a) separately for Tr and non-Tr descriptors. In the Tr case, we derive all the keys using Descriptor::<DescriptorPublicKey>::derived_descriptor which derives all the keys and throws away the conversion. Then separately it keeps around the un-derived descriptor, iterates through the keys, and populates the `tap_key_origins` map by re-computing the derivation. In the non-Tr case, we derive all the keys using the `KeySourceLookUp` object, which does exactly the same thing as `derived_descriptor` except that it stores its work in a BTreeMap, which is directly added to the PSBT's `item.bip32_derivation` field. This commit pulls out (a) into common code; it then copies all the data out of the key map into `item.tap_key_origins` along with an empty vector of tapleaves. It then goes through all the leaves, and for each key that appears in each leaf, appends that leaf's hash to the vector of tapleaves. This is still a little ineffecient but will be much cleaner after a later commit when we improve the Taproot SpendInfo structure. The original code dates to Lloyd's 2022 PR rust-bitcoin#339 which introduces logic to populate these maps. That algorithm underwent significant refactoring in response to review comments and I suspect that the duplicated logic went unnoticed after all the refactorings.

This commit introduces a new data structure but **does not** use it. The next commit will do this. I have separated them so that this one, which introduces a bunch of algorithmic code, can be reviewed separately from the API-breaking one. When computing a `Tr` output, we need to encode all its tapleaves into Script, put these into a Merkle tree and tweak the internal key with the root of this tree. When spending from one of the branches of this output, we need the Merkle path to that output. We currently do this by using the `TaprootSpendInfo` structure from rust-bitcoin. This is not a very good fit for rust-miniscript, because it constructs a map from Tapscripts to their control blocks. This is slow and memory-wasteful to construct, and while it makes random access fairly fast, it makes sequential access pretty slow. In Miniscript we almost always want sequential access, because all of our algorithms are some form of either "try every possibility and choose the optimum" or "aggregate every possibility". It also means that if there are multiple leaves with the same script, only one copy will ever be accessible. (If they are at different depths, the low-depth one will be yielded, but if they are at the same depth it's effectively random which one will get priority.) Having multiple copies of the same script is a pointless malleability vector, but this behavior is still surprising and annoying to have to think about. To replace `bitcoin::TaprootSpendInfo` we create a new structure `TrSpendInfo`. This structure doesn't maintain any maps: it stores a full Merkleized taptree in such a way that it can efficiently yield all of the leaves' control blocks in-order. It is likely that at some point we will want to upport this, or some variant of it, into rust-bitcoin, since for typical usecases it's much faster to use and construct.

See previous commit for details about this data structure. This commit stops using the rust-bitcoin `TaprootSpendInfo` in favor of our new `TrSpendInfo` structure. This one allows us to iterate over all the leaves of the Taptree, easily accessing their leaf hashes and control blocks in order, which simplifies satisfaction logic.

Moves a bit of ugly logic out of the PSBT module into the spendinfo module so that it's available for other users. We can convert from a TrSpendInfo to a bitcoin::TapTree but we can't do the opposite conversion since TrSpendInfo expects to have a Miniscript for each leaf.

This is a bit of a pain because the old lookup-by-script behavior doesn't map cleanly onto the new iterate-through-everything behavior. But this test is definitely useful (the unit tests in the previous commit came from it.)

apoelstra

On 3d12b42 successfully ran local tests

apoelstra · 2025-05-20T22:15:34Z

ping @sanket1729

sanket1729 · 2025-05-22T06:49:16Z

Reviewing, 2 commits in. Will continue with the rest tomorrow.

sanket1729

ACK 3d12b42

Thanks for adding the fixed test vectors, I cross checked that the same vectors pass before the changes in this PR.

Approving this, my feedback can be addressed in subsequent PRs.

sanket1729 · 2025-06-12T03:10:31Z

src/psbt/mod.rs

-                    tap_scripts.insert(control_block, leaf_script);
-                }

-                for (pk_pkh_derived, pk_pkh_xpk) in leaf_derived


I wonder why this was pk_pkh. Anycase, the new code is correct and much cleaner

sanket1729 · 2025-06-15T00:49:12Z

src/descriptor/tr/spend_info.rs

+    /// you need access to multiple control blocks at once, will likely need to clone and
+    /// store them separately.
+    #[inline]
+    pub fn control_block(&self) -> &ControlBlock { &self.control_block }


Why return &ControlBlock here if Item contains the contains ControlBlock. This is fine, but we might also want to give users into_control_block that consumes self?

TaprootMerkleBranch::try_from(merkle_stack) .expect("merkle stack guaranteed to be within allowable length")

I don't think there are any lifetime issues here because we allocate and copy the entire merkle branch into this structure?

Yeah, this seems reasonable. Will do in a followup.

sanket1729 · 2025-06-15T01:00:47Z

src/descriptor/tr/mod.rs

-        let spend_info = Arc::new(data);
-        *self.spend_info.lock().expect("Lock poisoned") = Some(Arc::clone(&spend_info));
-        spend_info
+        TrSpendInfo::from_tr(self)


I think you forgot to cache this inside the descriptor?

Oops, yep. Will fix this in a followup. I think maybe I had an earlier iteration of this PR that didn't cache it.

This was an oversight in rust-bitcoin#815.

bf10e45 compiler: pull non-binary argument errors from PolicyError into CompilerError (Andrew Poelstra) 3bc6c44 compiler: use Miniscript terminal constructors directly (Andrew Poelstra) b8d8910 miniscript: add infallible multi() and multi_a() constructors (Andrew Poelstra) fcbcf4e compiler: revert malleability check to simpler version (Andrew Poelstra) f98f263 compiler: forbid fragment probabilities of 0 (Andrew Poelstra) 16b4d60 fuzz/generate-files: support jujutsu (Andrew Poelstra) e000146 tr: cache TrSpendInfo when computing it (Andrew Poelstra) 2284858 tr: add `into_control_block` accessor to TrSpendInfoIterItem (Andrew Poelstra) Pull request description: This PR is a grab back of prepatory commits for larger changes. It includes a couple of followups to #815 in the first commits. Then it cleans up some error types and paths. I don't believe anything in here will be tricky or controversial, though there are API breaks because of the cleaned up error types. ACKs for top commit: sanket1729: ACK bf10e45 Tree-SHA512: 8a6c9006fb6473187a2ba69f0311fc29756c5f99544e35ab499658910e8f82a10eff740a19634b5d0c6807c583b8cd79fdc9fbf8a59b78ab96d5b387759db2b7

… `TaprootSpendInfo` with new miniscript-specific structure 3d12b42a6527bce318ac618d25dd25e2d2946470 fuzz: update taptree regression test to also check control blocks (Andrew Poelstra) c1d4305db39cf38be823c1e44fcd9580fba15cd9 tr: add a whole bunch of fixed vector unit tests for TrSpendInfo (Andrew Poelstra) 8bc940018b11ff3de18657cc8040f892a0ced652 tr: add conversion from TrSpendInfo to bitcoin::TapTree (Andrew Poelstra) 5fe2d25decd8b31f72692c5750a9d59c760c4c1c tr: replace `bitcoin::TaprootSpendInfo` with `TrSpendInfo`, update psbt (Andrew Poelstra) 795cd4fc8fd2228195d1d88a62cefc343eb5b04e tr: introduce new `TrSpendInfo` structure which holds a full TapTree (Andrew Poelstra) 27a30e61f7dd95f8719241b1add3438dd04aadce psbt: untangle some logic in `update_item_with_descriptor_helper` (Andrew Poelstra) 00495a5442ff0f66c71728e317084ddcc1ed2ff7 descriptor: add unit test from sanket (Andrew Poelstra) Pull request description: In Miniscript, to compute control blocks, estimate satisfaction costs, or otherwise iterate through all the leaves of a Taptree, we use the `bitcoin::TaprootSpendInfo` structure to maintain a map of all leaves. This map is inappropriate for Miniscript (it may not be appropriate for *anyone* actually..) for a few reasons: * It is a map from Tapleaves' encoding as Script to data about the Tapleaves; but in Miniscript the Script encoding isn't primary and isn't even available for non-`ToPublicKey` keys * This map structure means that if duplicate leaves exist then only one of the dupes will be accessible. * The map structure is also really inefficient; it stores the entire merkle path for each leaf even those these paths significantly overlap, leading to O(n log n) space instead of O(n). * Furthermore, we don't need *any* map because we only ever iterate through the entire tree. We fix all these issues by introducing a new `TrSpendInfo` struct which stores the entire Merkle tree in a flat representation and can produce an iterator over all the leaves. The iterator item can be used to access the Script, the Miniscript, the leaf version, and the control block for each leaf, while the `TrSpendInfo` structure itself can be used to access the internal and external keys. In other words, this one structure efficiently implements APIs for everything that rust-miniscript needs. This completes the Taproot API overhaul project. After this I'll go back to fixing error types, eliminating recursive structures, or overhauling the validation parameters, whichever one seems most tractable from the current state of `master`. ACKs for top commit: sanket1729: ACK 3d12b42a6527bce318ac618d25dd25e2d2946470 Tree-SHA512: 40fa71ebf22b104304ddbd9b3bf9efdd5d509c071a9461c80eb5f669ecdfd4dd7174cb5fcd2af30f32e45aecd370e077a49fd771eb71edf3f6fe4949c9b80d7d

apoelstra added 7 commits May 2, 2025 17:03

descriptor: add unit test from sanket

00495a5

tr: add a whole bunch of fixed vector unit tests for TrSpendInfo

c1d4305

apoelstra force-pushed the 2025-03--taproot-api branch from 700feab to 3d12b42 Compare May 2, 2025 17:07

apoelstra commented May 2, 2025

View reviewed changes

apoelstra mentioned this pull request May 18, 2025

It is basically impossible to get a LeafHash from a Tr descriptor to sign it #739

Open

apoelstra mentioned this pull request May 30, 2025

Top level checks are only used for miniscript APIs and not descriptor APIs. #788

Open

sanket1729 approved these changes Jun 15, 2025

View reviewed changes

apoelstra merged commit e5f50ab into rust-bitcoin:master Jun 15, 2025
31 checks passed

apoelstra deleted the 2025-03--taproot-api branch June 15, 2025 16:03

apoelstra added a commit to apoelstra/rust-miniscript that referenced this pull request Jun 20, 2025

tr: cache TrSpendInfo when computing it

d0613cf

This was an oversight in rust-bitcoin#815.

apoelstra mentioned this pull request Jun 20, 2025

Various cleanups, mostly around the compiler #828

Merged

apoelstra added a commit to apoelstra/rust-miniscript that referenced this pull request Jun 20, 2025

tr: cache TrSpendInfo when computing it

880f046

This was an oversight in rust-bitcoin#815.

apoelstra added a commit to apoelstra/rust-miniscript that referenced this pull request Jun 20, 2025

tr: cache TrSpendInfo when computing it

18402ee

This was an oversight in rust-bitcoin#815.

apoelstra added a commit to apoelstra/rust-miniscript that referenced this pull request Jul 8, 2025

tr: cache TrSpendInfo when computing it

e000146

This was an oversight in rust-bitcoin#815.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Taproot API Project] replace `TaprootSpendInfo` with new miniscript-specific structure #815

[Taproot API Project] replace `TaprootSpendInfo` with new miniscript-specific structure #815

Uh oh!

apoelstra commented May 2, 2025

Uh oh!

apoelstra left a comment

Uh oh!

apoelstra commented May 20, 2025

Uh oh!

sanket1729 commented May 22, 2025

Uh oh!

sanket1729 left a comment

Uh oh!

sanket1729 Jun 12, 2025

Uh oh!

sanket1729 Jun 15, 2025

Uh oh!

apoelstra Jun 15, 2025

Uh oh!

sanket1729 Jun 15, 2025

Uh oh!

apoelstra Jun 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[Taproot API Project] replace TaprootSpendInfo with new miniscript-specific structure #815

[Taproot API Project] replace TaprootSpendInfo with new miniscript-specific structure #815

Uh oh!

Conversation

apoelstra commented May 2, 2025

Uh oh!

apoelstra left a comment

Choose a reason for hiding this comment

Uh oh!

apoelstra commented May 20, 2025

Uh oh!

sanket1729 commented May 22, 2025

Uh oh!

sanket1729 left a comment

Choose a reason for hiding this comment

Uh oh!

sanket1729 Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

sanket1729 Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

apoelstra Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

sanket1729 Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

apoelstra Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[Taproot API Project] replace `TaprootSpendInfo` with new miniscript-specific structure #815

[Taproot API Project] replace `TaprootSpendInfo` with new miniscript-specific structure #815