multi: implement BIP 341 and 342 a.k.a complete taproot and tapscript consensus verification logic #1787

Roasbeef · 2022-01-07T02:58:39Z

In this PR, we implement full validation logic for BIP 341 and BIP 342. Along the way, we've created a series of utility functions to be used in wallet libraries to handle things like parsing the control block, generating merkle tapscript inclusion proofs, and computing the two sighash variants introduced in the BIPs.

See each commit message for a detailed description w.r.t the scope of the incremental changes, as well as notes for reviewers w.r.t improvements to the logic/routines or things to doubly verify.

This PR builds on a series of PRs including:

Steps to Graduate from a Draft PR

Merge in the BIP 340 implementation linked above, along with the new v2 module for btcec.
Merge in the modified BIP 9 implementation linked above, that implements the new min activation height vector.
Extract the generated test vectors from the bitcoind implementation (the BIPs contain no test vectors, but they have some good coverage saved from their fuzz testing) to ensure the logic implemented is compliant with consensus rules.
Add additional test coverage for the utillity methods included
Re-examine the current routine for generating the script tree, to examine more efficient variants, or variants that produce a non-leaning binary tree structure

Fixes everything in #1735, other than the btcwallet specific components.

guggero

Very excited about this PR, amazing work! Did a first quick pass, mostly to get a sense of the size of the diff. I have to say, it's not as large as I expected.
It'll probably grow quite a bit once tests with all test vectors are added.

txscript/script.go

txscript/engine.go

txscript/taproot.go

gcsfred2 · 2022-01-11T21:58:14Z

In script.go, calls to checkScriptParses have scriptVersion hardcoded to 0. Is that ok?

txscript/taproot.go

positiveblue

Really excited to see btcd supporting taproot! First pass done with a couple of small nits

txscript/standard.go

txscript/sigcache.go

txscript/opcode.go

wpaulino

Great work putting all of this together! I've completed an initial pass in which I mostly focused on spec compat.

The PR will become a bit unwieldy once tests have been added, I think it would be useful to break it up into smaller parts to help reviewers focus on some of the independent changes being added. An example breakdown could be (each point represents its own PR):

Prep work for Taproot integration
Taproot key spend path
Control block construction/verification
Taproot script spend path (including sig opcode changes)

txscript/hashcache.go

txscript/script.go

txscript/taproot.go

txscript/opcode.go

guggero

Started some end-to-end integrations (see https://github.com/guggero/btcwallet/tree/schnorr-taproot and https://github.com/guggero/lnd/tree/schnorr-taproot-signet) and found a few small-ish issues. Going to do a more in-depth review of this PR again tomorrow, just leaving those comments here.

txscript/taproot.go

txscript/hashcache.go

coveralls · 2022-02-21T00:22:32Z

Pull Request Test Coverage Report for Build 1962253109

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 79.676%

Totals
Change from base Build 1960508882:	0.0%
Covered Lines:	1231
Relevant Lines:	1545

💛 - Coveralls

Roasbeef · 2022-02-21T00:24:31Z

Just pushed a revamped version which addresses all existing review comments, and also commits the bitcoind taproot test vectors. With the latest set of commits, we pass all 2760 test cases (both success and reject sub-cases for each test case). I've now removed the "draft" label from this PR, so we can consider it to be more final-ish!

I have some smaller TODO's to clean up still, like better error messages, but I'm pretty confident/happy in the current diff. Still entertaining the option of splitting this up into smaller PRs as with the latest diff (ignore all the files committed as those are just test vectors) the PR size has swelled a bit.

3nprob · 2022-02-21T10:29:03Z

The little I've been looking through, this looks great!

Splitting up into smaller PRs would help a lot, if you do have the inclination ^^

guggero

Did a full review (third round) and was able to end-to-end test key spend, script (single leaf and multiple leaf) execution, including OP_CHECKSIG with schnorr signatures.

Found a few small issues but all in all I'd say this is very close to be ready! Given we have superb coverage of the validation code thanks to the test cases from bitcoind I'd say this can be merged very soon.

Really great work on this, the code is super clean and easy to read/understand!

txscript/standard.go

txscript/standard_test.go

txscript/sighash.go

txscript/taproot.go

txscript/sign.go

guggero · 2022-02-22T11:57:37Z

I fixed a few things I mentioned in the comments here: https://github.com/guggero/btcd/tree/1787-fixes feel free to incorporate directly into this PR.

Roasbeef · 2022-02-23T00:58:38Z

@guggero thanks! Added those in, and also working on more tests to ensure we accept sigs/witnesses we generate.

guggero

End-to-end tested this latest iteration in lnd and all test cases were successful. Of course more unit tests are always great to get more coverage on the signing part.

Since the verification part is covered very well, I'm giving my LGTM so we can move forward with the depending PRs.

wpaulino

This is looking pretty good! Very happy to see such an extensive testing suite for these changes.

I'll be working on a small PoC leveraging the new APIs to produce and verify tapscript spends and report back any findings in my next review.

txscript/pkscript.go

txscript/standard.go

blockchain/utxoviewpoint.go

txscript/hashcache.go

wpaulino · 2022-02-23T19:48:46Z

txscript/sigvalidate.go

+		opts...,
+	)
+	if err != nil {
+		// TODO(roasbeef): propagate the error here?


Perhaps worth cross-referencing with what bitcoind does in this case?

calcTaprootSignatureHashRaw only returns en error if:

An invalid sighash is used, bitcoind does the same here: https://github.com/bitcoin/bitcoin/blob/623745ca74cf3f54b474dac106f5802b7929503f/src/script/interpreter.cpp#L1716

An index is passed in that's greater than the total number of actual inputs

For this, earlier in their pipeline there's an assertion to fail early for this case. We have a return clause here as we expose this function to consumers of the library, which may pass in an invalid index.

The other error paths there are just error returns from the io.Writer when creating the sig message, which in practice don't return an error.

txscript/sigvalidate.go

wpaulino · 2022-02-23T21:38:56Z

txscript/taproot.go

+	// remaining bytes to be a multiple of the node size, which is 32
+	// bytes.
+	case (len(ctrlBlock)-ControlBlockBaseSize)%ControlBlockNodeSize != 0:
+		return nil, fmt.Errorf("invalid max block size")


Nit: this is the same error text as above.

Will resolve these in a final pass to insert proper errors.

txscript/taproot.go

txscript/engine.go

Roasbeef · 2022-02-25T02:07:18Z

Responded to the latest round of comments! Also added a new CalcTapscriptSignaturehash that can be used to compute the sighash of tapscript spends. This'll be useful when doing things like validating that the remote party sent a valid HTLC signature on the LN context.

On the lnd side, we've successfully done end-to-end tests of both keyspend and script spend paths (using this lnd PR and the btcwallet PR linked).

In this commit, we use the recently added checksig verifiers to validate signatures for pre-segwit, and segwit v0 scripts.

In this commit, we add two new functions: one for signing a raw top-level taproot keyspend, and another for generating a valid witness for a keyspend.

In this commit, we add the initial verification logic for top-level taproot keyspends. Keyspends use the base BIP 341 sighash digest and don't require any tapscript level functionality for validation.

We'll use this to examine if a script has any OP_SUCCESS op codes during pre-processing before we attempt full tapscript execution.

In this commit, we add a new struct to represent the ControlBlock structure used to feed in the tapscript leaf inclusion proof into the witness tack. The `ParseControlBlock` parses a would-be control block and returns an error if it's incorrectly formatted.

In this commit, we add a new function to verify the taproot merkle commitment of a given tapscript leaf. Along the way we add some helper functions which can be used to construct a taproot output given the raw script root.

In this commit, we add a new function `RawTxInTapscriptSignature` that will be used to generate signatures in the _tapscript_ context. Note that this differs from top-level taproot as a distinct sighash is used, and we _always_ accept a root hash to perform the proper tweak.

…sses In this commit, we add a new AssembleTaprootScriptTree function that given a list of tapscript leaves, generates a valid tapscript root, along with the auxiliary proof data needed to spend each output.

In this commit, we use the recently added control block and script tree verification+generation routines to implement full script path verification within the VM. This includes verifying the script reveal commitment, and recursing one layer deeper to execute the revealed witness script as specified by BIP 342.

…script

… position We'll need this to properly generate the sighash during tapscript validation later

In this commit, we implement the new checksig semantics as part of tapscript validation. Namely: * OP_CHECKSIGVERIFY no longer pops the OP_TRUE off the stack (TODO(roasbeef): verify)) * the new sig ops semantics are added where each sig deducts 50 from a starting budget of 50+the weight of the witness * NULLFAIL is always enforced, meaning invalid sigs MUST be an empty sig array

In this commit, we implement OP_CHECKSIGADD which replaces OP_CHECKMULTISIG* in the tapscript execution environment.

In this commit, we add a total of 2760 taproot reference tests generated by the bitcoind functional tests located at: https://github.com/bitcoin/bitcoin/blob/master/test/functional/feature_taproot.py. The tests aren't deterministic (fresh private keys are generated), so we time we go to update the set of tests, we'll end up with fresh hashes (the file name is the sha1 of the raw json test) and tests.

In this commit, we add the deployment parameters of taproot as specified in the deployment section of BIp 341: https://github.com/bitcoin/bips/blob/master/bip-0341.mediawiki#deployment. Take note of the custom activation threshold, as well as the specified min activation heights for mainnet only.

This PR includes some changes to them, so we'll need to use a temporary replace directives to ensure the build passes.

Roasbeef · 2022-03-16T01:30:31Z

Mainnet sync ✅

🚨 Just received word from Merge Control: this PR is a go!! 🚨

Roasbeef added soft forks taproot labels Jan 7, 2022

Roasbeef mentioned this pull request Jan 7, 2022

txscript: add taproot script type #1768

Closed

guggero reviewed Jan 11, 2022

View reviewed changes

txscript/script.go Outdated Show resolved Hide resolved

txscript/script.go Outdated Show resolved Hide resolved

txscript/script.go Outdated Show resolved Hide resolved

txscript/engine.go Show resolved Hide resolved

txscript/taproot.go Outdated Show resolved Hide resolved

gcsfred2 reviewed Jan 18, 2022

View reviewed changes

txscript/taproot.go Outdated Show resolved Hide resolved

positiveblue reviewed Feb 2, 2022

View reviewed changes

txscript/standard.go Outdated Show resolved Hide resolved

txscript/sigcache.go Show resolved Hide resolved

txscript/opcode.go Show resolved Hide resolved

wpaulino reviewed Feb 3, 2022

View reviewed changes

Roasbeef force-pushed the taproot-impl branch from da170c1 to e0d13b7 Compare February 9, 2022 01:03

guggero reviewed Feb 14, 2022

View reviewed changes

txscript/taproot.go Outdated Show resolved Hide resolved

txscript/taproot.go Outdated Show resolved Hide resolved

guggero mentioned this pull request Feb 14, 2022

Taproot: integrate btcec/v2 and btcwallet changes to support Taproot key spend paths in lnd's wallet lightningnetwork/lnd#6263

Merged

Roasbeef mentioned this pull request Feb 15, 2022

multi: update signing RPCs to expose new tapscript+taproot sig types lightningnetwork/lnd#6267

Closed

3 tasks

alexbosworth reviewed Feb 15, 2022

View reviewed changes

txscript/hashcache.go Outdated Show resolved Hide resolved

Roasbeef force-pushed the taproot-impl branch from 52006f8 to c7a457a Compare February 16, 2022 01:18

guggero mentioned this pull request Feb 16, 2022

Taproot: Add taproot address and signing capabilities btcsuite/btcwallet#792

Merged

Roasbeef force-pushed the taproot-impl branch from c7a457a to dd86580 Compare February 21, 2022 00:13

Roasbeef marked this pull request as ready for review February 21, 2022 00:14

Roasbeef force-pushed the taproot-impl branch from dd86580 to e95aafc Compare February 21, 2022 00:20

guggero requested changes Feb 22, 2022

View reviewed changes

Roasbeef force-pushed the taproot-impl branch 2 times, most recently from 417425c to f5186a5 Compare February 23, 2022 00:58

guggero approved these changes Feb 23, 2022

View reviewed changes

wpaulino reviewed Feb 23, 2022

View reviewed changes

Roasbeef force-pushed the taproot-impl branch from f5186a5 to 4629cbd Compare February 25, 2022 02:05

Roasbeef added 22 commits March 15, 2022 18:22

txscript: use new signature verifiers for existing CHECKSIG ops

1ac34b7

In this commit, we use the recently added checksig verifiers to validate signatures for pre-segwit, and segwit v0 scripts.

txscript: add new functions for signing a top-level taproot output

938c193

In this commit, we add two new functions: one for signing a raw top-level taproot keyspend, and another for generating a valid witness for a keyspend.

txscript: add VM verification logic for top-level taproot keyspends

2ac743d

In this commit, we add the initial verification logic for top-level taproot keyspends. Keyspends use the base BIP 341 sighash digest and don't require any tapscript level functionality for validation.

txscript: use keyBytes instead of ScriptHash for segwit utilities

a7c3db4

txscript: add new ScriptHasOpSuccess utility method

11dd820

We'll use this to examine if a script has any OP_SUCCESS op codes during pre-processing before we attempt full tapscript execution.

txscript: add VerifyTaprootLeafCommitment function

37f8c8b

In this commit, we add a new function to verify the taproot merkle commitment of a given tapscript leaf. Along the way we add some helper functions which can be used to construct a taproot output given the raw script root.

txscript: add AssembleTaprootScriptTree func for creating input witne…

17e4609

…sses In this commit, we add a new AssembleTaprootScriptTree function that given a list of tapscript leaves, generates a valid tapscript root, along with the auxiliary proof data needed to spend each output.

txscript: disable OP_CHECKMULTISIG and OP_CHECKMULTISIGVERIFY for tap…

a7a8ad7

…script

txscript: always enforce MINIMAL_IF during tapscript execution

3c6be73

txscript: add new OpcodePosition method to tokenizer to save code sep…

a4beed9

… position We'll need this to properly generate the sighash during tapscript validation later

txscript: implement OP_CHECKSIGADD

5f8660e

In this commit, we implement OP_CHECKSIGADD which replaces OP_CHECKMULTISIG* in the tapscript execution environment.

blockchain: use taproot script flags for validation after activation

23cf18b

blockchain/indexers: add P2TR support to the addrindex

f7f7bb3

blockchain: fix IsSpeedy() bug, add more logging in BIP 9 state machine

6ab97a3

txscript: add more detailed taproot errors

99e4e00

build: add temporary replace directives for btcec+chainhash

30d58b9

This PR includes some changes to them, so we'll need to use a temporary replace directives to ensure the build passes.

Roasbeef force-pushed the taproot-impl branch from f1f53d1 to 30d58b9 Compare March 16, 2022 01:24

Roasbeef merged commit fc36cb2 into btcsuite:master Mar 16, 2022

Roasbeef mentioned this pull request Mar 16, 2022

multi: taproot mega tracking issue #1735

Closed

15 tasks

arshbot mentioned this pull request Mar 21, 2022

Initial Schnorr signature support #1667

Closed

louisinger mentioned this pull request May 31, 2022

Taproot support vulpemventures/go-elements#195

Closed

This was referenced Jun 10, 2022

Taproot support vulpemventures/go-elements#198

Merged

Taproot support vulpemventures/go-elements#199

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multi: implement BIP 341 and 342 a.k.a complete taproot and tapscript consensus verification logic #1787

multi: implement BIP 341 and 342 a.k.a complete taproot and tapscript consensus verification logic #1787

Roasbeef commented Jan 7, 2022 •

edited

Loading

guggero left a comment

gcsfred2 commented Jan 11, 2022

positiveblue left a comment

wpaulino left a comment

guggero left a comment

coveralls commented Feb 21, 2022 •

edited

Loading

Roasbeef commented Feb 21, 2022

3nprob commented Feb 21, 2022

guggero left a comment

guggero commented Feb 22, 2022

Roasbeef commented Feb 23, 2022

guggero left a comment

wpaulino left a comment

wpaulino Feb 23, 2022

Roasbeef Feb 25, 2022 •

edited

Loading

wpaulino Feb 23, 2022

Roasbeef Feb 25, 2022

Roasbeef commented Feb 25, 2022

Roasbeef commented Mar 16, 2022

multi: implement BIP 341 and 342 a.k.a complete taproot and tapscript consensus verification logic #1787

multi: implement BIP 341 and 342 a.k.a complete taproot and tapscript consensus verification logic #1787

Conversation

Roasbeef commented Jan 7, 2022 • edited Loading

Steps to Graduate from a Draft PR

guggero left a comment

Choose a reason for hiding this comment

gcsfred2 commented Jan 11, 2022

positiveblue left a comment

Choose a reason for hiding this comment

wpaulino left a comment

Choose a reason for hiding this comment

guggero left a comment

Choose a reason for hiding this comment

coveralls commented Feb 21, 2022 • edited Loading

Pull Request Test Coverage Report for Build 1962253109

💛 - Coveralls

Roasbeef commented Feb 21, 2022

3nprob commented Feb 21, 2022

guggero left a comment

Choose a reason for hiding this comment

guggero commented Feb 22, 2022

Roasbeef commented Feb 23, 2022

guggero left a comment

Choose a reason for hiding this comment

wpaulino left a comment

Choose a reason for hiding this comment

wpaulino Feb 23, 2022

Choose a reason for hiding this comment

Roasbeef Feb 25, 2022 • edited Loading

Choose a reason for hiding this comment

wpaulino Feb 23, 2022

Choose a reason for hiding this comment

Roasbeef Feb 25, 2022

Choose a reason for hiding this comment

Roasbeef commented Feb 25, 2022

Roasbeef commented Mar 16, 2022

Roasbeef commented Jan 7, 2022 •

edited

Loading

coveralls commented Feb 21, 2022 •

edited

Loading

Roasbeef Feb 25, 2022 •

edited

Loading