[segnet3 hard-fork] Allow arbitrary commitment tree structure.

[maaku]

The witness root is allowed to be placed at an arbitrary position up to seven layers deep in a Merkle tree structure. The witness nonce is now the branch through the commitment tree to the witness root, and a single byte is added to the commitment output specifying this path in compact form. This allows other consensus commitments to be added in the future with a minimal number of bytes and without requiring a certain position for each commitment in the tree.

This hard-forks segnet3 as it changes the witness commitment structure.

[sipa]

How does this prevent creating a block that commits to two different witnesses?

[sipa]

Never mind, I misread, that branch byte goes into the commitment.

So in a future with 8 consensus-critical commitments, you would now have
32_3 bytes in the witness nonce, and similarly 32_3 bytes in each of the 7
other ones. Plus, you'd need to store the branch bytes for each of them
somewhere. As they're consensus critical commitments, all fully validating
nodes need all them anyway, so the tree approach is less efficient than the
chain approach (and that remains the case even when using a maximally
unbalanced tree).

[maaku]

This is correct. It is a tradeoff though because the tree based approach means that as the number of commitments grows the proof size does not grow linearly.

More importantly, it doesn't set in stone the position of the commitment in the tree. We are free to reorganize (as policy) the tree as we see fit going forward. Miners also are allowed to prioritize commitments as they see fit, such as merged mining commitments which benefit greatly from being near the top of the tree without having a setaside in the bitcoin consensus rules.

[sipa]

Mixing consensus-critical and non-consensus critical commitments would significantly worsen adoption, I'm afraid, as one is influenced by the transaction selection code, and the other chosen by miners. Combining them means that we need protocols through the whole mining stack that pass up the commitments to be made and combine them. Your tree approach is close to what I implemented first, but it seems that splitting consensus-critical commitments and non-consensus-critical ones (using a future tree based approach, committed separately) is far simpler. And once you drop the requirement of non-consensus critical commitments, the advantage of using a tree disappears (except for slightly worsening fraud proofs, but that will need a hardfork anyway to improve).

[maaku]

A more efficient tree encoding is possible that doesn't repeat hashes. I am open to implementing it if there is desire. It only comes into play if there are multiple commitments. This was just easier to implement and illustrate the point.

This may not be obvious, but not that the structure of this coinbase witness containing the hashes / nonce is protocol level, not consensus and is upgradeable.

[sipa]

I know a more efficient mechanism is possible, and I'm interested in helping develop it. But I think we need a separate, simpler, construction for consensus-critical commitments. And for that goal, any tree is less efficient.

[jl2012]

This is more or less the original design (you can find it in the history of BIP141). Consensus critical commitment are by definition verified by every full node, even it might be obsoleted in the future. Therefore, a linear structure will guarantee the highest efficiency. It also simplifies the implementation a lot.

The only case where a tree structure might be needed is when we have 2 parallel softforks proposals with new consensus critical commitment. But we can always introduce a tree structure any time as the witness nonce is flexible.

[maaku]

@jl2012 I believe that to be taking too narrow a view of what these commitments are useful for. These commitments allow construction of various compact proofs of consensus state. These proofs are in turn useful for various space sensitive applications, of which I will highlight two: proofs of state for mobile wallets, and cross-chain smart contracting such as the two-way peg. Both of these applications are highly space constrained -- in the wallet case because mobile data is expensive, slow, and/or power hungry, and the cross-chain smart contracting case because these proofs must be broadcast-able on the bitcoin or bitcoin-like block chains. Using a tree structure for commitments allows tradeoffs to be made favouring smaller proofs for the more space-constrained applications, and larger proofs for those which are only checked by the consensus code or very rarely needed.

To take a concrete example, one of the very first commitments that is likely to be added is a hash of a constrained nonce + the previous block in order to prevent validationless mining. This is likely to be merged well prior to any sort of txout or block header commitment schemes. Yet because the linear structure forces an otherwise arbitrary time-based ordering of commitments, any proof using a later commitment would necessarily have to include 32 bytes of unnecessary information just to validate. And if we're talking about things like SMS fraud proof relay or on-chain two-way peg withdraw proofs, that 32 bytes could be VERY expensive.

To your other point, this is derivative of the original commitment structure indeed, because it was a suggested improvement to the original code back in December. I have only had time now to work on it and was surprised to see what was in my view a regression in functionality. This is an argument for development happening as a pull-request against the bitcoin/bitcoin repo, where the PR itself can act as a sort of mailing list for coordinating development. Right now things are not being communicated well. But that is an aside.

[sipa]

Any Merkle path to a commitment will already require the full coinbase transaction (~160 bytes) and path inside the main Merkle tree (~350 bytes). Yes, a linear structure is slightly worse for proofs on further commitments, but if 32 bytes hurts them, they're likely already impossible.

[jl2012]

@maaku If the size of SPV proof is really that critical, we probably should consider a hardfork. Coinbase tx > 5kB is not uncommon. If the required proof is in a big coinbase tx, obviously you can't use SMS to relay it.

For things like previous block proof or UTXO commitment, which do not have their own data structure, may not be able to join the witness commitment at all (no matter in a linear or a tree structure; unless they are deployed at the same time with segwit). It is because the Merkle path for such commitments will have no place to go. We need either a new witness structure just for storing a particular Merkle path (which is an overkill), or put the path in the coinbase tx. Then why don't we simply commit them directly in the coinbase tx?

This problem could be probably solved this way: instead of using 0x0000....0000 as the coinbase witness ID, we could use the hash of the coinbase witness data in the witness commitment. Therefore, we can have an arbitrary size for coinbase witness, instead of restricting it to 32 bytes. So the Merkle path of future commitments can go there.

[sipa]

The ideas here interact with Peter Todd's suggestion to allow unvalidated data inside block witnesses, instead of using further P2P changes for later commitments.

If we're going to do an incompatible change, can you have a look at http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-February/012313.html ?

[sipa]

The discussion above focusses on the distinction between two different paths to take (C = number of commitments, T = number of transactions, B = coinbase size):

• A linear structure, which needs 32 * C full node overhead, and 32 * C + 32 * log2(T) + B worst-case fraud proof overhead.
• A tree structure, which needs 32 * log2(C) * C full node overhead, and 32_log2(C) + 32_log2(T) + B fraud proof overhead.

For any low number of C, I don't think it matters much. The size of fraud proofs will be dominated by the coinbase transaction and transaction merkle path. We need a fundamentally different approach (and presumably a hard fork) to change that.

Also, neither choice actually forces us to keep following the same path down the line. We can add other commitment locations to the transaction tree, or add extra commitments to the existing commitment coinbase output scriptPubKey. In the linear case, we can create a tree commitment that gets stored inside the coinbase witness later (which is just p2p) and commit it elsewhere. In the tree case, we can commit to a linear sequence in one of the leaves.