Merge pull request #4278 from PastaPastaPasta/backports-0.18-pr9

Backports 0.18 pr9

doc/README.md

@@ -76,6 +76,7 @@ The Dash Core repo's [root README](/README.md) contains relevant information on
 - [Tor Support](tor.md)
 - [Init Scripts (systemd/upstart/openrc)](init.md)
 - [ZMQ](zmq.md)
+- [PSBT support](psbt.md)
 
 License
 ---------------------

doc/psbt.md

@@ -0,0 +1,132 @@
+# PSBT Howto for Bitcoin Core
+
+Since Bitcoin Core 0.17, an RPC interface exists for Partially Signed Bitcoin
+Transactions (PSBTs, as specified in
+[BIP 174](https://github.com/bitcoin/bips/blob/master/bip-0174.mediawiki)).
+
+This document describes the overall workflow for producing signed transactions
+through the use of PSBT, and the specific RPC commands used in typical
+scenarios.
+
+## PSBT in general
+
+PSBT is an interchange format for Bitcoin transactions that are not fully signed
+yet, together with relevant metadata to help entities work towards signing it.
+It is intended to simplify workflows where multiple parties need to cooperate to
+produce a transaction. Examples include hardware wallets, multisig setups, and
+[CoinJoin](https://bitcointalk.org/?topic=279249) transactions.
+
+### Overall workflow
+
+Overall, the construction of a fully signed Bitcoin transaction goes through the
+following steps:
+
+- A **Creator** proposes a particular transaction to be created. They construct
+  a PSBT that contains certain inputs and outputs, but no additional metadata.
+- For each input, an **Updater** adds information about the UTXOs being spent by
+  the transaction to the PSBT. They also add information about the scripts and
+  public keys involved in each of the inputs (and possibly outputs) of the PSBT.
+- **Signers** inspect the transaction and its metadata to decide whether they
+  agree with the transaction. They can use amount information from the UTXOs
+  to assess the values and fees involved. If they agree, they produce a
+  partial signature for the inputs for which they have relevant key(s).
+- A **Finalizer** is run for each input to convert the partial signatures and
+  possibly script information into a final `scriptSig` and/or `scriptWitness`.
+- An **Extractor** produces a valid Bitcoin transaction (in network format)
+  from a PSBT for which all inputs are finalized.
+
+Generally, each of the above (excluding Creator and Extractor) will simply
+add more and more data to a particular PSBT, until all inputs are fully signed.
+In a naive workflow, they all have to operate sequentially, passing the PSBT
+from one to the next, until the Extractor can convert it to a real transaction.
+In order to permit parallel operation, **Combiners** can be employed which merge
+metadata from different PSBTs for the same unsigned transaction.
+
+The names above in bold are the names of the roles defined in BIP174. They're
+useful in understanding the underlying steps, but in practice, software and
+hardware implementations will typically implement multiple roles simultaneously.
+
+## PSBT in Bitcoin Core
+
+### RPCs
+
+- **`converttopsbt` (Creator)** is a utility RPC that converts an
+  unsigned raw transaction to PSBT format. It ignores existing signatures.
+- **`createpsbt` (Creator)** is a utility RPC that takes a list of inputs and
+  outputs and converts them to a PSBT with no additional information. It is
+  equivalent to calling `createrawtransaction` followed by `converttopsbt`.
+- **`walletcreatefundedpsbt` (Creator, Updater)** is a wallet RPC that creates a
+  PSBT with the specified inputs and outputs, adds additional inputs and change
+  to it to balance it out, and adds relevant metadata. In particular, for inputs
+  that the wallet knows about (counting towards its normal or watch-only
+  balance), UTXO information will be added. For outputs and inputs with UTXO
+  information present, key and script information will be added which the wallet
+  knows about. It is equivalent to running `createrawtransaction`, followed by
+  `fundrawtransaction`, and `converttopsbt`.
+- **`walletprocesspsbt` (Updater, Signer, Finalizer)** is a wallet RPC that takes as
+  input a PSBT, adds UTXO, key, and script data to inputs and outputs that miss
+  it, and optionally signs inputs. Where possible it also finalizes the partial
+  signatures.
+- **`finalizepsbt` (Finalizer, Extractor)** is a utility RPC that finalizes any
+  partial signatures, and if all inputs are finalized, converts the result to a
+  fully signed transaction which can be broadcast with `sendrawtransaction`.
+- **`combinepsbt` (Combiner)** is a utility RPC that implements a Combiner. It
+  can be used at any point in the workflow to merge information added to
+  different versions of the same PSBT. In particular it is useful to combine the
+  output of multiple Updaters or Signers.
+- **`decodepsbt`** is a diagnostic utility RPC which will show all information in
+  a PSBT in human-readable form, as well as compute its eventual fee if known.
+
+### Workflows
+
+#### Multisig with multiple Bitcoin Core instances
+
+Alice, Bob, and Carol want to create a 2-of-3 multisig address. They're all using
+Bitcoin Core. We assume their wallets only contain the multisig funds. In case
+they also have a personal wallet, this can be accomplished through the
+multiwallet feature - possibly resulting in a need to add `-rpcwallet=name` to
+the command line in case `bitcoin-cli` is used.
+
+Setup:
+- All three call `getnewaddress` to create a new address; call these addresses
+  *Aalice*, *Abob*, and *Acarol*.
+- All three call `getaddressinfo X`, with *X* their respective address, and
+  remember the corresponding public keys. Call these public keys *Kalice*,
+  *Kbob*, and *Kcarol*.
+- All three now run `addmultisigaddress 2 ["Kalice","Kbob","Kcarol"]` to teach
+  their wallet about the multisig script. Call the address produced by this
+  command *Amulti*. They may be required to explicitly specify the same
+  addresstype option each, to avoid constructing different versions due to
+  differences in configuration.
+- They also run `importaddress "Amulti" "" false` to make their wallets treat
+  payments to *Amulti* as contributing to the watch-only balance.
+- Others can verify the produced address by running
+  `createmultisig 2 ["Kalice","Kbob","Kcarol"]`, and expecting *Amulti* as
+  output. Again, it may be necessary to explicitly specify the addresstype
+  in order to get a result that matches. This command won't enable them to
+  initiate transactions later, however.
+- They can now give out *D* as address others can pay to.
+
+Later, when *V* BTC has been received on *Amulti*, and Bob and Carol want to
+move the coins in their entirety to address *Asend*, with no change. Alice
+does not need to be involved.
+- One of them - let's assume Carol here - initiates the creation. She runs
+  `walletcreatefundedpsbt [] {"Asend":V} 0 false {"subtractFeeFromOutputs":[0], "includeWatching":true}`.
+  We call the resulting PSBT *P*. P does not contain any signatures.
+- Carol needs to sign the transaction herself. In order to do so, she runs
+  `walletprocesspsbt P`, and gives the resulting PSBT *P2* to Bob.
+- Bob inspects the PSBT using `decodepsbt "P2"` to determine if the transaction
+  has indeed just the expected input, and an output to *Asend*, and the fee is
+  reasonable. If he agrees, he calls `walletprocesspsbt "P2"` to sign. The
+  resulting PSBT *P3* contains both Carol's and Bob's signature.
+- Now anyone can call `finalizepsbt "P2"` to extract a fully signed transaction
+  *T*.
+- Finally anyone can broadcast the transaction using `sendrawtransaction "T"`.
+
+In case there are more signers, it may be advantageous to let them all sign in
+parallel, rather passing the PSBT from one signer to the next one. In the
+above example this would translate to Carol handing a copy of *P* to each signer
+separately. They can then all invoke `walletprocesspsbt P`, and end up with
+their individually-signed PSBT structures. They then all send those back to
+Carol (or anyone) who can combine them using `combinepsbt`. The last two steps
+(`finalizepsbt` and `sendrawtransaction`) remain unchanged.

src/bech32.cpp

@@ -58,7 +58,7 @@ uint32_t PolyMod(const data& v)
     // v, it corresponds to x^2 + v0*x + v1 mod g(x). As 1 mod g(x) = 1, that is the starting value
     // for `c`.
     uint32_t c = 1;
-    for (auto v_i : v) {
+    for (const auto v_i : v) {
         // We want to update `c` to correspond to a polynomial with one extra term. If the initial
         // value of `c` consists of the coefficients of c(x) = f(x) mod g(x), we modify it to
         // correspond to c'(x) = (f(x) * x + v_i) mod g(x), where v_i is the next input to
@@ -149,7 +149,7 @@ std::string Encode(const std::string& hrp, const data& values) {
     data combined = Cat(values, checksum);
     std::string ret = hrp + '1';
     ret.reserve(ret.size() + combined.size());
-    for (auto c : combined) {
+    for (const auto c : combined) {
         ret += CHARSET[c];
     }
     return ret;

src/bench/mempool_eviction.cpp

@@ -16,7 +16,7 @@ static void AddTx(const CMutableTransaction& tx, const CAmount& nFee, CTxMemPool
     bool spendsCoinbase = false;
     unsigned int sigOpCost = 4;
     LockPoints lp;
-    pool.addUnchecked(tx.GetHash(), CTxMemPoolEntry(
+    pool.addUnchecked(CTxMemPoolEntry(
                                         MakeTransactionRef(tx), nFee, nTime, nHeight,
                                         spendsCoinbase, sigOpCost, lp));
 }

src/cuckoocache.h

@@ -396,15 +396,15 @@ class cache
         std::array<uint32_t, 8> locs = compute_hashes(e);
         // Make sure we have not already inserted this element
         // If we have, make sure that it does not get deleted
-        for (uint32_t loc : locs)
+        for (const uint32_t loc : locs)
             if (table[loc] == e) {
                 please_keep(loc);
                 epoch_flags[loc] = last_epoch;
                 return;
             }
         for (uint8_t depth = 0; depth < depth_limit; ++depth) {
             // First try to insert to an empty slot, if one exists
-            for (uint32_t loc : locs) {
+            for (const uint32_t loc : locs) {
                 if (!collection_flags.bit_is_set(loc))
                     continue;
                 table[loc] = std::move(e);
@@ -468,7 +468,7 @@ class cache
     inline bool contains(const Element& e, const bool erase) const
     {
         std::array<uint32_t, 8> locs = compute_hashes(e);
-        for (uint32_t loc : locs)
+        for (const uint32_t loc : locs)
             if (table[loc] == e) {
                 if (erase)
                     allow_erase(loc);

src/dash-cli.cpp

@@ -46,7 +46,7 @@ static void SetupCliArgs()
     gArgs.AddArg("-named", strprintf("Pass named instead of positional arguments (default: %s)", DEFAULT_NAMED), false, OptionsCategory::OPTIONS);
     gArgs.AddArg("-rpcclienttimeout=<n>", strprintf("Timeout in seconds during HTTP requests, or 0 for no timeout. (default: %d)", DEFAULT_HTTP_CLIENT_TIMEOUT), false, OptionsCategory::OPTIONS);
     gArgs.AddArg("-rpcconnect=<ip>", strprintf("Send commands to node running on <ip> (default: %s)", DEFAULT_RPCCONNECT), false, OptionsCategory::OPTIONS);
-    gArgs.AddArg("-rpccookiefile=<loc>", _("Location of the auth cookie. Relative paths will be prefixed by a net-specific datadir location. (default: data dir)"), false, OptionsCategory::OPTIONS);
+    gArgs.AddArg("-rpccookiefile=<loc>", "Location of the auth cookie. Relative paths will be prefixed by a net-specific datadir location. (default: data dir)", false, OptionsCategory::OPTIONS);
     gArgs.AddArg("-rpcpassword=<pw>", "Password for JSON-RPC connections", false, OptionsCategory::OPTIONS);
     gArgs.AddArg("-rpcport=<port>", strprintf("Connect to JSON-RPC on <port> (default: %u or testnet: %u)", defaultBaseParams->RPCPort(), testnetBaseParams->RPCPort()), false, OptionsCategory::OPTIONS);
     gArgs.AddArg("-rpcuser=<user>", "Username for JSON-RPC connections", false, OptionsCategory::OPTIONS);

src/httprpc.h

@@ -23,7 +23,7 @@ void StopHTTPRPC();
 /** Start HTTP REST subsystem.
  * Precondition; HTTP and RPC has been started.
  */
-bool StartREST();
+void StartREST();
 /** Interrupt RPC REST subsystem.
  */
 void InterruptREST();

src/init.cpp

@@ -530,7 +530,7 @@ void SetupServerArgs()
 
     gArgs.AddArg("-addressindex", strprintf("Maintain a full address index, used to query for the balance, txids and unspent outputs for addresses (default: %u)", DEFAULT_ADDRESSINDEX), false, OptionsCategory::INDEXING);
     gArgs.AddArg("-reindex", "Rebuild chain state and block index from the blk*.dat files on disk", false, OptionsCategory::INDEXING);
-    gArgs.AddArg("-reindex-chainstate", "Rebuild chain state from the currently indexed blocks", false, OptionsCategory::INDEXING);
+    gArgs.AddArg("-reindex-chainstate", "Rebuild chain state from the currently indexed blocks. When in pruning mode or if blocks on disk might be corrupted, use full -reindex instead.", false, OptionsCategory::INDEXING);
     gArgs.AddArg("-spentindex", strprintf("Maintain a full spent index, used to query the spending txid and input index for an outpoint (default: %u)", DEFAULT_SPENTINDEX), false, OptionsCategory::INDEXING);
     gArgs.AddArg("-timestampindex", strprintf("Maintain a timestamp index for block hashes, used to query blocks hashes by a range of timestamps (default: %u)", DEFAULT_TIMESTAMPINDEX), false, OptionsCategory::INDEXING);
     gArgs.AddArg("-txindex", strprintf("Maintain a full transaction index, used by the getrawtransaction rpc call (default: %u)", DEFAULT_TXINDEX), false, OptionsCategory::INDEXING);
@@ -1038,8 +1038,7 @@ static bool AppInitServers()
     StartRPC();
     if (!StartHTTPRPC())
         return false;
-    if (gArgs.GetBoolArg("-rest", DEFAULT_REST_ENABLE) && !StartREST())
-        return false;
+    if (gArgs.GetBoolArg("-rest", DEFAULT_REST_ENABLE)) StartREST();
     StartHTTPServer();
     return true;
 }

src/net.cpp

@@ -381,7 +381,7 @@ CNode* CConnman::FindNode(const CService& addr, bool fExcludeDisconnecting)
 bool CConnman::CheckIncomingNonce(uint64_t nonce)
 {
     LOCK(cs_vNodes);
-    for (CNode* pnode : vNodes) {
+    for (const CNode* pnode : vNodes) {
         if (!pnode->fSuccessfullyConnected && !pnode->fInbound && pnode->GetLocalNonce() == nonce)
             return false;
     }
@@ -2019,7 +2019,7 @@ void CConnman::ThreadDNSAddressSeed()
 
             LOCK(cs_vNodes);
             int nRelevant = 0;
-            for (auto pnode : vNodes) {
+            for (const CNode* pnode : vNodes) {
                 nRelevant += pnode->fSuccessfullyConnected && !pnode->fFeeler && !pnode->fOneShot && !pnode->m_manual_connection && !pnode->fInbound && !pnode->m_masternode_probe_connection;
             }
             if (nRelevant >= 2) {
@@ -2126,7 +2126,7 @@ int CConnman::GetExtraOutboundCount()
     int nOutbound = 0;
     {
         LOCK(cs_vNodes);
-        for (CNode* pnode : vNodes) {
+        for (const CNode* pnode : vNodes) {
             // don't count outbound masternodes
             if (pnode->m_masternode_connection) {
                 continue;
@@ -2201,7 +2201,7 @@ void CConnman::ThreadOpenConnections(const std::vector<std::string> connect)
         std::set<std::vector<unsigned char> > setConnected;
         if (!Params().AllowMultipleAddressesFromGroup()) {
             LOCK(cs_vNodes);
-            for (CNode* pnode : vNodes) {
+            for (const CNode* pnode : vNodes) {
                 if (!pnode->fInbound && !pnode->m_masternode_connection && !pnode->m_manual_connection) {
                     // Netgroups for inbound and addnode peers are not excluded because our goal here
                     // is to not use multiple of our limited outbound slots on a single netgroup

src/net_processing.cpp

@@ -1163,7 +1163,7 @@ void PeerLogicValidation::BlockConnected(const std::shared_ptr<const CBlock>& pb
     // Erase orphan transactions included or precluded by this block
     if (vOrphanErase.size()) {
         int nErased = 0;
-        for (uint256 &orphanHash : vOrphanErase) {
+        for (const uint256& orphanHash : vOrphanErase) {
             nErased += EraseOrphanTx(orphanHash);
         }
         LogPrint(BCLog::MEMPOOL, "Erased %d orphan tx included or conflicted by block\n", nErased);

src/qt/bitcoingui.cpp

@@ -1839,7 +1839,7 @@ void UnitDisplayStatusBarControl::mousePressEvent(QMouseEvent *event)
 void UnitDisplayStatusBarControl::createContextMenu()
 {
     menu = new QMenu(this);
-    for (BitcoinUnits::Unit u : BitcoinUnits::availableUnits())
+    for (const BitcoinUnits::Unit u : BitcoinUnits::availableUnits())
     {
         QAction *menuAction = new QAction(QString(BitcoinUnits::name(u)), this);
         menuAction->setData(QVariant(u));

src/qt/dash.cpp

@@ -423,7 +423,7 @@ void BitcoinApplication::requestShutdown()
 
 #ifdef ENABLE_WALLET
     window->removeAllWallets();
-    for (WalletModel *walletModel : m_wallet_models) {
+    for (const WalletModel* walletModel : m_wallet_models) {
         delete walletModel;
     }
     m_wallet_models.clear();

src/qt/peertablemodel.cpp

@@ -68,7 +68,7 @@ class PeerTablePriv
             node.getNodesStats(nodes_stats);
             cachedNodeStats.reserve(nodes_stats.size());
 
-            for (auto& node_stats : nodes_stats)
+            for (const auto& node_stats : nodes_stats)
             {
                 CNodeCombinedStats stats;
                 stats.nodeStats = std::get<0>(node_stats);

src/qt/sendcoinsdialog.cpp

@@ -422,7 +422,7 @@ void SendCoinsDialog::send(QList<SendCoinsRecipient> recipients)
     questionString.append("<hr />");
     CAmount totalAmount = currentTransaction.getTotalTransactionAmount() + txFee;
     QStringList alternativeUnits;
-    for (BitcoinUnits::Unit u : BitcoinUnits::availableUnits())
+    for (const BitcoinUnits::Unit u : BitcoinUnits::availableUnits())
     {
         if(u != model->getOptionsModel()->getDisplayUnit())
             alternativeUnits.append(BitcoinUnits::formatHtmlWithUnit(u, totalAmount));

src/qt/splashscreen.cpp

@@ -204,7 +204,7 @@ void SplashScreen::unsubscribeFromCoreSignals()
 #ifdef ENABLE_WALLET
     m_handler_load_wallet->disconnect();
 #endif // ENABLE_WALLET
-    for (auto& handler : m_connected_wallet_handlers) {
+    for (const auto& handler : m_connected_wallet_handlers) {
         handler->disconnect();
     }
     m_connected_wallet_handlers.clear();

src/qt/transactiondesc.cpp

@@ -164,13 +164,13 @@ QString TransactionDesc::toHTML(interfaces::Node& node, interfaces::Wallet& wall
     else
     {
         isminetype fAllFromMe = ISMINE_SPENDABLE;
-        for (isminetype mine : wtx.txin_is_mine)
+        for (const isminetype mine : wtx.txin_is_mine)
         {
             if(fAllFromMe > mine) fAllFromMe = mine;
         }
 
         isminetype fAllToMe = ISMINE_SPENDABLE;
-        for (isminetype mine : wtx.txout_is_mine)
+        for (const isminetype mine : wtx.txout_is_mine)
         {
             if(fAllToMe > mine) fAllToMe = mine;
         }

src/qt/transactionrecord.cpp

@@ -82,14 +82,14 @@ QList<TransactionRecord> TransactionRecord::decomposeTransaction(interfaces::Wal
     {
         bool involvesWatchAddress = false;
         isminetype fAllFromMe = ISMINE_SPENDABLE;
-        for (isminetype mine : wtx.txin_is_mine)
+        for (const isminetype mine : wtx.txin_is_mine)
         {
             if(mine & ISMINE_WATCH_ONLY) involvesWatchAddress = true;
             if(fAllFromMe > mine) fAllFromMe = mine;
         }
 
         isminetype fAllToMe = ISMINE_SPENDABLE;
-        for (isminetype mine : wtx.txout_is_mine)
+        for (const isminetype mine : wtx.txout_is_mine)
         {
             if(mine & ISMINE_WATCH_ONLY) involvesWatchAddress = true;
             if(fAllToMe > mine) fAllToMe = mine;

src/rest.cpp

@@ -460,7 +460,7 @@ static bool rest_getutxos(HTTPRequest* req, const std::string& strURIPart)
                 oss >> fCheckMemPool;
                 oss >> vOutPoints;
             }
-        } catch (const std::ios_base::failure& e) {
+        } catch (const std::ios_base::failure&) {
             // abort in case of unreadable binary data
             return RESTERR(req, HTTP_BAD_REQUEST, "Parse error");
         }
@@ -634,11 +634,10 @@ static const struct {
       {"/rest/blockhashbyheight/", rest_blockhash_by_height},
 };
 
-bool StartREST()
+void StartREST()
 {
     for (unsigned int i = 0; i < ARRAYLEN(uri_prefixes); i++)
         RegisterHTTPHandler(uri_prefixes[i].prefix, false, uri_prefixes[i].handler);
-    return true;
 }
 
 void InterruptREST()