elements: Add support of Liquid-BTC on elements #3078

cdecker · 2019-09-20T18:24:03Z

This has been left to stew for quite some time, and I think I'm finally happy
with the way it turned out (that is to say "I'm getting tired of having to
rebase it since it touches a lot of code" 😉). I rebase and squashed it
from the state it was in for the announcement, and fixed a couple of more
fee-estimation bugs (we do estimate fees in a lot of different places it turns
out), and now we no longer get crashes running the test-suite (yay!).

We do however still get a couple of failures, most likely due to the test
being very bitcoin-specific (assuming bitcoin tx sizes for fees, and using
regtest addresses). I'm slowly making my way through them and parametrizing as
many as I can with sensible effort, but some of them I will just mark as
skippable since making them chainparam-dependent can be a lot of work.

The following tests still need triaging:

The tests were run with valgrind in the elements configuration, and no
issues were found. I won't add that configuration to Travis-CI though since we
are far better off with adding that configuration to our own CI system once we
have it running.

There is one really ugly part that I don't like about this PR and that's the
use of the is_elements global variable. Since we have started passing the
chainparams through to all things handling transactions, we might be able to
remove that in the future, but I'll do that in a separate (less
conflict-prone) PR, so please don't judge me too hard on that one, I was young
and inexperienced 😉

And yes, I know its a huge PR, sorry 🙍‍♂️

rustyrussell

Generally really good, obviously some fixups need squashing.

I think you should make chainparams a global in utils, as you did for plugins. This is better than is_elements as you'll get a nice crash if it isn't set. Probably do that and all the "wire into init msgs" in one commit. We can leave chainparams inside txs as a future aspiration if we ever want to dp multichain. That could easily go in as a separate PR, if we wanted.

rustyrussell · 2019-09-23T00:15:55Z

common/utils.c

@@ -6,6 +6,7 @@

 secp256k1_context *secp256k1_ctx;
 const tal_t *tmpctx;
+bool is_elements = false;


Worse, this is not initialized in subdaemons; defaulting to false seem wrong. Classic trick here is to use a pointer, which nicely segfaults if not initialized/

rustyrussell · 2019-09-23T00:17:43Z

bitcoin/tx.c

+
+	/* For whatever reason the length computation gets upset if we tell it
+	 * that we are using elements. It wants to discover it on its own, NO
+	 * CLUES! (Ms. Doyle) */


Inline a reference to the Elements issue?

I filed ElementsProject/libwally-core#139 for this an fixed up the comment 👍

rustyrussell · 2019-09-23T00:22:43Z

bitcoin/block.c

+		push_le32(b->hdr.target, push, &p);
+		push_le32(b->hdr.nonce, push, &p);
+	}
+	sha256_double(&out->shad, p, tal_bytelen(p));


Creating an array for this is nasty. You want:

struct sha256_ctx shactx; sha256_init(&shactx); sha256_update(...); sha256_le32(...)... sha256_double_done(&shactx, &out->shad);

rustyrussell · 2019-09-23T00:23:44Z

bitcoin/tx.c

@@ -97,6 +97,9 @@ const u8 *bitcoin_tx_output_get_script(const tal_t *ctx,
 	u8 *res;
 	assert(outnum < tx->wtx->num_outputs);
 	output = &tx->wtx->outputs[outnum];
+	if (output->features & WALLY_TX_IS_COINBASE)
+		return NULL;


This may be invalid for elements, but it's certainly possible for bitcoin?

You also need to sweep all the other callers of bitcoin_tx_output_get_script: there are quite a few, and they don't expect NULL.

Made it a bit more generic: if libwally hands us back a NULL script, we return a NULL script. That way we defer the decision to libwally.

I also went through all callers and added NULL-handling where not already done so.

lightningd/chaintopology.c

bitcoin/tx.c

rustyrussell · 2019-09-23T00:50:53Z

tests/test_misc.py

+
+
+def test_minconf_withdraw(node_factory, bitcoind):
+    """Issue 2518: ensure that ridiculous confirmation levels don't overflow


How did this new test get in here?

rustyrussell · 2019-09-23T00:51:07Z

tests/test_misc.py

@@ -452,6 +453,187 @@ def is_p2wpkh(output):
    assert only_one(fundingtx['vin'])['txid'] == res['wallettxid']


+def test_withdraw(node_factory, bitcoind, chainparams):


How did this new test get in here?

rustyrussell · 2019-09-23T00:53:53Z

wallet/wallet.c

@@ -420,6 +424,9 @@ static const struct utxo **wallet_select(const tal_t *ctx, struct wallet *w,
 		/* Account for witness (1 byte count + sig + key) */
 		input_weight += 1 + (1 + 73 + 1 + 33);

+		/* Elements inputs have 6 bytes of blank proofs attached. */
+		input_weight += 6;
+


But you didn't make this conditional on elements?

rustyrussell · 2019-09-23T00:55:41Z

bitcoin/tx.c

@@ -384,7 +384,7 @@ struct bitcoin_tx *bitcoin_tx(const tal_t *ctx,
 	/* If we are constructing an elements transaction we need to
 	 * explicitly add the fee as an extra output. So allocate one more
 	 * than the outputs we need internally. */
-	if (is_elements)
+	if (chainparams->is_elements)


This commit message is not correct any more?

Fixed up into a later cleanup that does this kind of conversion.

cdecker · 2019-09-25T22:44:19Z

Still working on the remaining review comments, will rebase and squash tomorrow.

cdecker · 2019-09-26T17:08:17Z

Ok, mostly fixed up the feedback. Still working on the non-fee-paying assets being set to 0. I'll add multi-asset support for tracking (but only consider the fee-paying asset in our UTXO selection).