$1500 bounty for the creation of address index in btcd

[ivanaszuber]

The Counterparty team is offering a $1500 bounty for the creation of address index in btcd.

Counterparty is currently using local Insight servers for providing counterpartyd with Bitcoin block explorer functionality, and we'd like to switch to using btcd as our backend as soon as it's production-ready. For this we need btcd to index transactions by address.

Related: #185

[davecgh]

To get anyone willing to take this on started, here are some key points about what needs to happen for this to become a reality:

• The btcdb interface will likely need to have functions added to expose the new information. This in turn will mean modifying the backend btcdb driver ldb and memdb to implement those new interface functions.
• The functionality will have to be optional and provide a new flag on btcd which allows the user to enable it. As mentioned elsewhere, I'd really like to see this capability available, but it will be expensive in terms of disk space, so it will need to be opt-in.
• Since the ability will be optional, it will need the ability to track the last known good block for the index and concurrently build the index accordingly. Some scenarios:
  • The block height is already at the latest block height when the option is enabled and has never been enabled before -- start the build of the index from block 0
  • The block height client has the address index built already up to block height 150,000 and btcd is restarted without the optional parameter -- it should no longer continue building the address index, but it should not delete what it has already built
  • The block height client has the address index built already up to block height 150,000 and btcd is restarted with the optional parameter -- it should continue building the address index where it left off
• Any RPC calls to obtain this obtain this information will either need to fail if the address index isn't built to the current block yet, or include the block it's built through
• Not required, but I suspect a way to optionally delete the address index would be nice

[Roasbeef]

This looks pretty interesting. I might be willing to take a crack at it. I'm pretty familiar with the btcd codebase after submitting a few minor contributions, and working on some of my own experiments using btcd.

First off, I have a few questions (some of which discussed briefly w/ @davecgh on irc earlier):

• Does the final implementation require an implementation of the addr index for both the ldb and memdb interfaces implementations? I'm assuming only leveldb?
• Also, should the completed RPC interface emulate that of searchrawtransactions, in jmcorgan's addrindex fork of Bitcoin Core? Or are there any additional requirements or separate RPC calls required?
• Finally, @ivanaszuber i'm assuming the bounty would be payed in BTC, right? Ideally, could we suspend the funds in a multi-sig (2-of-3) escrow? Perhaps secured between representatives from Counterparty, and Conformal?

Thanks for the great overview @davecgh . I think I have a good picture of how to best integrate this into btcd, staying true to code style and concurrency patterns throughout the codebase. Throughout development, I'll be sure to maintain open communication with both parties to ensure all are satisfied with the work.

If all is agreed upon, I may be able to start as early as next week. I have two completely free weeks coming up, and should be able to make a major development push towards completion.

[ouziel-slama]

@Roasbeef, that sound great!

Also, should the completed RPC interface emulate that of searchrawtransactions, in jmcorgan's addrindex fork of Bitcoin Core? Or are there any additional requirements or separate RPC calls required?

yeah the interface should emulate searchrawtransactions but ideally with only one difference: it should accept a second optional boolean parameter unconfirmed. If this parameter is set to True, searchrawtransactions should also return unconfirmed transactions.

[ivanaszuber]

Finally, @ivanaszuber i'm assuming the bounty would be payed in BTC, right? Ideally, could we suspend the funds in a multi-sig (2-of-3) escrow? Perhaps secured between representatives from Counterparty, and Conformal?

@Roasbeef sure, let's do that, if @davecgh agrees

[ouziel-slama]

@Roasbeef, you can also avoid the additional parameter and just include unconfirmed transactions with the field confirmations: 0.

[davecgh]

@Roasbeef As mentioned on IRC, in regads to memdb, I think you should only need the stubs to make the interface compile.

I'm fine with being one of the the escrow holders.

[Roasbeef]

@ouziel-slama confirmed, the API call will return the number of confirmations for each transaction (with zero begin unconfirmed)

@ivanaszuber and @davecgh who will be the third signatory for the 2-of-3 escrow? Also to flesh out the details a bit more, what will be the agreed upon protocol for the multi-sig itself. Pass around a partially signed transaction, use something like Bitpay's Copay? Only thing required on my part is, (through maybe a more private channel), presenting an address to pay to, correct?

I feel like we should discuss the remainder of the escrow and payment details, "offline" via a more private channel. I can use this issue for development updates/questions once I've began.

[davecgh]

@Roasbeef I would assume another person from the Counterparty team since they are offering the bounty. Either that, or go with a 1-of-2 so either @ivanaszuber or me can release the funds. I figured we'd just use a partially signed transaction. Your address is only needed when the funds are released. Prior to that, the involved escrow holders create the multisig transaction using our pubkeys and send the funds to the resulting p2sh address (3xxxxxxx) where they remain in escrow until they are eventually released by spending them to an address you provide. I'm available on IRC to discuss particulars.

[adamkrellenstein]

The traditional way of handling this situation, AFAIK, is to use 2-of-3, where @Roasbeef himself holds one key. That way neither @ivanaszuber nor @davecgh can run away with the funds, and @Roasbeef gets paid if either @ivanaszuber or @davecgh thinks he deserves to be.

[davecgh]

@adamkrellenstein Good point. That makes the most sense.

[adamkrellenstein]

If @davecgh and @Roasbeef e-mail their public keys to @ivanaszuber (ivanaz@counterparty.io), we can send the funds to a CHECKMULTISIG address composed of theirs and her own.

[davecgh]

Done

[ivanaszuber]

Thanks @davecgh , got it.

[Roasbeef]

Done

[ivanaszuber]

@Roasbeef and @davecgh sent you an email with transaction details

[carryforward]

I was asking about this back in May 2014 for one of my earlier BTC address tracking applications. The use case goes way beyond Counterparty transactions. I will also be very useful for Factom as well, since we will be relying on the Bitcoin blockchain for security.

[Roasbeef]

@carryforward great to hear others will also have a use for the new feature.

The final PR has been marked as ready for review. I'm still currently testing the current code, trying to find and shake out any lingering bugs. I've indexed the testnet (for speed of index) chain a few times over, and exercised all the possible scenarios w.r.t to syncing vs building the index. All seems to be functionally correct, but I'd also appreciate any help testing out the new changes.

$ btcctl searchrawtransaction mrvtPm91tQyRH968D53bW6G9VuhSuMBfjj
[
    {
        "hex": "01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff0e037cd60402ef06062f503253482fffffffff01fe2f059500000000232103df0bd541bdb6966d0f4a0c25e17cb26306bbf772cfa6860d3081b3836e8b53aeac00000000",
        "txid": "73f6f95bed4ac47fa0b8985ff1952ce3549161f4aa2cc43874824de9ed8bcad1",
        "version": 1,
        "locktime": 0,
        "vin": [
            {
                "coinbase": "037cd60402ef06062f503253482f",
                "sequence": 4294967295
            }
        ],
        "vout": [
            {
                "value": 25.0014515,
                "n": 0,
                "scriptPubKey": {
                    "asm": "03df0bd541bdb6966d0f4a0c25e17cb26306bbf772cfa6860d3081b3836e8b53ae OP_CHECKSIG",
                    "hex": "2103df0bd541bdb6966d0f4a0c25e17cb26306bbf772cfa6860d3081b3836e8b53aeac",
                    "reqSigs": 1,
                    "type": "pubkey",
                    "addresses": [
                        "mrvtPm91tQyRH968D53bW6G9VuhSuMBfjj"
                    ]
                }
            }
        ],
        "blockhash": "0000000088b4fc96db56725d84fa3da717ba6070b2d701ba4b3969ea29849871",
        "confirmations": 1,
        "time": 1420581689,
        "blocktime": 1420581689
    }
]

[davecgh]

@ivanaszuber, @adamkrellenstein: PR #205 is nearing completion. I'm still doing a lot of testing on it and it needs another round of code review, but it should be ready for you guys to pull and test to make sure it does what is expected. As @Roasbeef posted above, once the address index is activated with --addrindex and has caught up to the main chain, the new searchrawtransaction RPC will return all transactions which involve the provided address.

EDIT: I should note that it takes quite a while to build the index since it has to load, parse, and extract the addresses from every single script in every transaction in the block chain.

[ivanaszuber]

Thanks @davecgh & @Roasbeef ! We'll start our testing soon and let you know if we run into any issues.

[ivanaszuber]

@davecgh we've done our initial testing and it seems to work as expected (we didn't run into any issues so far). Once you complete your code review we can proceed with the payment

[davecgh]

@ivanaszuber Thanks for the update. I'm done with the code review and testing. The only thing that remains is for the searchrawtransactions bool -> int fix and then I'll get it merged.

[davecgh]

This has been implemented by #205 and is now available on master.