searchrawtransactions -- vin does not include address.

[dan-da]

vout includes an array of addresses but vin does not.

Other APIs I have used/evaluated include the address field whenever an address is present (ie non coinbase transactions). For example, here is output from toshi.io and btcd for the same address

https://bitcoin.toshi.io/api/v0/addresses/1GeBuz6M42h5mhWmSDFhmXsdNjdqNn6D76/transactions

...
      "inputs":[
        {
          "previous_transaction_hash":"2494a070c8f2749b00e60267d0c7e7a41a160b4069d88c6cc6dfa8e96eb0dfbb",
          "output_index":1,
          "amount":16621000000,
          "script":"30450220591ee83540e23a87728bb57a21db78b04cc10ff395b72b9e1d82baa30087d657022100809c1b1c62f227c7c7b5e034649b67cccc9ab81d90aba9da5b53df5e3a2b78e601 04f92c4e08d928ce95512e97212a5b2392dcf8f58030f6a1f519e541b9ea934ffbcfa5e1881517450aacabd48e26064d2a513b30cb54fa545876bd68f51709f7f1",
          "addresses":[
            "1GeBuz6M42h5mhWmSDFhmXsdNjdqNn6D76"
          ]
        }
      ],
...

And here is btcd output:

btcctl searchrawtransactions 1GeBuz6M42h5mhWmSDFhmXsdNjdqNn6D76

...
    "vin": [
      {
        "txid": "2494a070c8f2749b00e60267d0c7e7a41a160b4069d88c6cc6dfa8e96eb0dfbb",
        "vout": 1,
        "scriptSig": {
          "asm": "30450220591ee83540e23a87728bb57a21db78b04cc10ff395b72b9e1d82baa30087d657022100809c1b1c62f227c7c7b5e034649b67cccc9ab81d90aba9da5b53df5e3a2b78e601 04f92c4e08d928ce95512e97212a5b2392dcf8f58030f6a1f519e541b9ea934ffbcfa5e1881517450aacabd48e26064d2a513b30cb54fa545876bd68f51709f7f1",
          "hex": "4830450220591ee83540e23a87728bb57a21db78b04cc10ff395b72b9e1d82baa30087d657022100809c1b1c62f227c7c7b5e034649b67cccc9ab81d90aba9da5b53df5e3a2b78e6014104f92c4e08d928ce95512e97212a5b2392dcf8f58030f6a1f519e541b9ea934ffbcfa5e1881517450aacabd48e26064d2a513b30cb54fa545876bd68f51709f7f1"
        },
        "sequence": 4294967295
      }
...

My application needs the address from vin when available. So the impact on myself is that I either must drag in a large external library to parse the vin script or fork btcd and implement it there.

[dan-da]

I just noticed that the amount is not included in vin either, which I also need.

[luke-jr]

Inputs do not have addresses...

[dan-da]

agreed that inputs do not have addresses. nor amounts.

but the previous outputs that the inputs reference typically (always?) do have them. And applications sometimes need them. I suppose this is why most blockchain oracle type APIs I have looked at include both addresses and amounts for inputs. I provided a sample of toshi's output above.

It is orders of magnitude more efficient for the API provider to lookup the previous outputs and return addresses and amount inline than for an application to make individual getrawtransaction queries to lookup each previous transaction and find the matching outputs. For my particular application, it is the difference between useful and too slow to use.

Or if there is some other efficient way to obtain all coin in/out amounts for a particular address (all time) via the RPC API, please point me to it. My app is displaying a transaction history with running balance for one (or more) addresses.

anyway, I have been hacking on the btcd code and I got something working. I plan to submit a pull request soon(ish). Hopefully it (or an improved version) can be integrated.

One possibility: I could make the extra fields optional for:
a) purity
b) avoid extra lookups when not needed/wanted.

Another possibility would be to make it a separate API.

We can continue the discussion in the pull request when it is ready.

[luke-jr]

The address of an output that created a UTXO is not at all related to that UTXO's use as an input.

[dan-da]

1. Are you saying that toshi and other blockchain API's that present this information are wrong?
2. What solution do you recommend for application developers?

[davecgh]

As luke-jr pointed out, the "issue" here is that inputs do not have addresses.

When a transaction creates an unspent transaction output (UTXO), it might be paying directly to a pubkey, to a pubkey-hash (Bitcoin) address, to a bare multi-sig addresses, to a pay-to-script-hash (P2SH) address, or any other number of nonstandard uses. So, when a transaction input consumes one of those UTXOs, there is no guarantee that it has an address associated with it. In practice it usually does, but therein lies the problem - it's not foolproof and thus your application needs to be extremely careful to avoid being exploitable as a result.

When an address index is built, it is possible to track all of the funds coming in and out of a particular address, and that is what the APIs you mention are doing. @Roasbeef could probably speak to how difficult it would be for btcd to add that information when the address index is enabled since he's the most familiar with the address index format given he added the code for it.

In the mean time, what will probably work for you is the rescan API in btcd's websocket API. That allows you to specify an address and it delivers all transactions which pay coins to that address and consume coins from it thereby letting your application manually create a balance.

[luke-jr]

@dan-da Yes, anything presenting the previous sent-to address as a "from address" is wrong. The only solution is to use a unique address for each payment (which is the intended design in the first place).

@davecgh Even when the UTXO script was created by sending to an address, that address is entirely unrelated to the UTXO itself or the transaction spending it. The address's lifetime necessarily ends when the transaction is received.

[dan-da]

Hi Dave, thanks for your comments. I understand that transaction outputs do not always pay directly to address(es). However, often/usually they do, and it can be useful to provide address info in those cases as well as the amount of each prev output.

In the event that there is no address in the previous output then the addresses array can simply be empty or not present, or multiple addresses in the case of multisig -- as other APIs do. Here is a toshi multisig example.

https://bitcoin.toshi.io/api/v0/transactions/e8ac451e7f47d566d4dd822a67fea2181ecfa7c7ba96d9faa63e2f6b26e55fd3

  "inputs":[
    {
      "previous_transaction_hash":"...",
      "output_index":0,
      "amount":50000000,
      "script":"...",
      "addresses":[
        "1A8JiWcwvpY7tAopUkSnGuEYHmzGYfZPiq",
        "1AJbsFZ64EpEfS5UAjAfcUG8pH8Jn3rn1F",
        "1ETBbsHPvbydW7hGWXXKXZ3pxVh3VFoMaX"
      ]
    }
  ],

The point that neither of the comments have addressed is that other blockchain APIs are already doing this because it is useful. I already provided a link and sample output from toshi.io. I'll provide some more at the bottom.

Thanks for the link to the rescan API docs. I wasn't aware of that. Unfortunately I don't think it will help my use case for a couple reasons: 1) I am using the RPC API and changing to websockets isn't really an option at present. 2) rescan sounds very slow, and I need quick results. Hence my joy at finding the address index in btcd.

The searchrawtransactions API only functions when the address index is enabled. I have already wrestled with the code enough that I have the desired new functionality. I just need to a bit of cleanup and some more testing before publishing a pull request. Perhaps @Roasbeef can review it. Since there seems to be resistance to this functionality, I'll go ahead and make the new behavior optional.

As you can see below, the industry has already pretty much agreed on this. All these APIs include address and amount/value. btcd wouldn't exactly be breaking new ground here.

Bitpay Insight: https://insight.bitpay.com/api/addrs/18xKdc2kXMJYCz12QpEqTZXUpJympcKchY/txs

                        "vin": [
                                {
                                        "txid": "...",
                                        "vout": 1,
                                        "scriptSig": {
                                                "asm": "...",
                                                "hex": "..."
                                        },
                                        "sequence": 4294967295,
                                        "n": 0,
                                        "addr": "16y6iQDD1a7uVm6UvWJ7KdhHjGCJ5deMSx",
                                        "valueSat": 849990000,
                                        "value": 8.4999,
                                        "doubleSpentTxID": null
                                }
                        ],

blockchain.info:
https://blockchain.info/address/18xKdc2kXMJYCz12QpEqTZXUpJympcKchY?format=json

note: I like how they use a key named "prev_out" to make that clear.

   "inputs":[
      {
         "sequence":4294967295,
         "prev_out":{
            "spent":true,
            "tx_index":54282905,
            "type":0,
            "addr":"16y6iQDD1a7uVm6UvWJ7KdhHjGCJ5deMSx",
            "value":849990000,
            "n":1,
            "script":"76a9144174b1422c4ab89e1ce41ae4d71ffc78957cf53188ac"
         },

BlockCypher: http://dev.blockcypher.com/#address-full-endpoint

"inputs": [
        {
        "prev_hash": "729f6469b59fea5da7...",
        "output_index": 0,
        "script": "483045022100d06cdad1a...",
        "output_value": 3500000,
        "sequence": 4294967295,
        "addresses": [
            "1VxsEDjo6ZLMT99dpcLu4RQonMDVEQQTG"
        ],
        "script_type": "pay-to-pubkey-hash"
        },
        ...
    ],

Blocktrail: https://www.blocktrail.com/api/docs#api_data

 "inputs": [
                {
                    "index": 0,
                    "output_hash": "b0fc639ef699e8521dd4863fc9b8f6224d8aafb3e4fe52bb784150973c711523",
                    "output_index": 1,
                    "value": 105139900,
                    "address": "1qMBuZnrmGoAc2MWyTnSgoLuWReDHNYyF",
                    "type": "pubkeyhash",
                    "multisig": null,
                    "script_signature": "0x48 0x3045022100b7dcc897c7baadf693865818ea1fa7c544073936cad37f27fa402cb517989f7002201c56fb93e3acb3c016d394765f0a248cb85fa6ae290336fef88ca811dd6d6d0f01 0x41 0x04bc6b933acf9ff99e9c2946ecfe927a3813e0218bcfa58f3b5107a956a307055faa2609b879e51dda158820f93c8a22d12abfaa46c99d43e3ecc4c497c27fec5b"
                }
            ],

[Roasbeef]

Hey @dan-da, so it wouldn't be too difficult to add the functionality you desire to the current API for searchrawtransactions. However, you should be mindful of the comments made above by dave and luke-jr, although the API's you listed do indeed display "from addresses", the very existence of that terminology creates incorrect thought models which lead to people developing skewed models of the way Bitcoin actually works, which may directly lead to loss of user funds (eg: sending a refund to a "from address").

So back to the required changes. I'm unsure what you've done so far but, you shouldn't need to alter much concerning the address index itself. It's pretty minimal and just stores the info to identify the location of a particular hash160 inside a block. Plus, adding additional data to the index would considerably increase the final index size, which IMO is undesirable. It seems what you desire is showing full information about all previous outputs, rather than just the txid:vout combo. You'll have to alter either btcjson.TxRawResult or database.TxListReplay or create a variant of either which includes full information about the prev_out. Either way you'll need to perform an additional DB query to fetch the information for the referenced prev_out then embed it within your new response type (if you don't want to increase the size of the addr index).

Whatever changes you make concerning the addr index or the related searchrawtransactions API, I'll be happy to review (time permitting).

EDIT: accidentally sent out this comment prematurely before I had finished drafting it

[davecgh]

@luke-jr: Agreed about the lifetime. I had already posted more about that on the mailing list in regards to this issue. I'll quote it here for reference:

All of that said, the real underlying issue here is that there really is
no notion of a "from" address in Bitcoin. This is a constant source of
confusion for developers and users. An address is better thought of as
something like the unique ID of a purchase order or receipt. When
something is sold for an amount of BTC, a unique address should be
generated which uniquely identifies that single transaction (just like a
unique ID on a purchase order or receipt). The vendor knows a user has
paid once that unique address receives the agreed upon amount of funds
since the address is never again used for anything else. As you can
see, in this correct usage model, the "from" address is 100% irrelevant
since it's the unique address that identifies the specific order versus
trying to divine who paid by using a realistically non-existent "from"
address to identify the person making the payment.

The address index (and other techniques) are essentially hacks that are
often used to shoehorn this incorrect notion of a "from" address model
into the system.

[davecgh]

This has been implemented by #487.