eth_listTransactions

[tymat]

This feature has been discussed on gitter in the past and we would like to see this get implemented for accounts and contracts.

Proposal

eth_listTransactions

Returns a list of transaction hashes for a given external or contract account

Parameters

String - the account or contract address

params: [
  "0x385acafdb80b71ae001f1dbd0d65e62ec2fff055"
]

Returns

DATA - A list of transaction hashes

[gavofyork]

This is useless - transactions are merely one way of two ways to send a message to an object ("contract") in Ethereum; CALL would be the other way.

Use logs. It's what they were designed for.

[gavofyork]

would close.

[Gustav-Simonsson]

@gavofyork It's far from useless. For example, it would be very useful for keeping track of non-contract txs, which cannot use logs.

This feature has been requested from several people. We shouldn't force users to only use the platform in a certain way, and If users desire a API call which doesn't conflict with anything or have any significant drawbacks, we should consider adding it.

I'm in favour of adding this.

[obscuren]

So am I. But let the community decide on this. Get enough +1 and it will eventually be considered/added.

[ethernomad]

Maybe messages to non-contract addresses should automatically get a blank log entry? It would store the originating transaction id.

[frozeman]

👍 I fully agree with gustav here. Even though most of the cool stuff can happen in contracts, being able to simply send money from one account to the other will be heavily used. And then you need a easy way to display its transactions (e.g. the wallet needs that for example :) )

I would add a second parameter, which is the nonce, from which on you want to get transactions. This way you don't need to request the whole list over and over again.

[tymat]

eth_listTransactions is very much needed for Ether to Ether transactions. For wallet devs we need a way to present this information to users so they know how their Ether was spent and when it was spent and which TX hashes they occurred in. Adding a separate persistence layer to store this data for easy access is cumbersome and would have to be constantly updated on the client side.

This is very useful and much needed.

[slothbag]

+1 I found myself looking for this feature.. would be great to get it included.

[VoR0220]

👍

[VoR0220]

this is much more efficient IMO than the logs...and there's a lot of use cases for this feature for execution in other contracts.

[VoR0220]

I would add a third optional parameter for number of transactions from the top or bottom (perhaps something python-esque?)

[frozeman]

Though i +1 this feature @VoR0220 you can't get logs only from transactions, as there can also be logs created when one contract interacts with another.. So only relying on this feature won't work.

I don't like the third parameter actually, as a nonce (from which nonce to get tx on), is more than enough to control the tx flow.

[VoR0220]

how does nonce control how many transactions you would like to see?

[d11e9]

👍

[frozeman]

If you say i want to get all transactions from nonce 50 and there a re a total of 60 tx, you will get 10

[bas-vk]

@frozeman, what if you want to retrieve tx [100...150] when there are 10.000+ transactions?

[VoR0220]

@frozeman but that's for a nonce in the chain correct? So I would only get the transactions for that block, no (I might be totally off here and feel free to correct me)? However, if I wanted to go and see the past 50 or so transactions for a particular address, how does that nonce benefit me if I want to get specific with the transactions displayed?

[frozeman]

nonce is an ear increasing number of transactions from an account.
With web3.eth.getTransactionCount('0x12345..') you basically get the next used nonce (as nonces start from 0, like array.length)

And then you can subtract whatever number you want to go back and ask for those tx.

[frozeman]

@bas-vk you couldn't :)

[obscuren]

While easy to implement its difficult to do it right. Indexing all transactions would require quite significant storage and most certainly doesn't scale.

One way to do this is as how we MIP mapped LOGs and use a similar query mechanism, but this is an expensive and CPU/IO intensive operation because there are far more transactions than there are LOGs.

Doing this for LOGs is actually fine since you more or less paid for it where as transactions incurs no additional cost in terms of upfront payment. letting the user decide what to map isn't an option either since it just won't work for light clients. Light clients depend on archive and full nodes for data retrieval and can't make use of this "user setting" option because they don't receive any other data than the data that is explicit requested.

I understand this is a much requested feature but understand that it comes at a great, expensive cost and, as I said earlier, difficult to do it right.

[frozeman]

I don't expect dapps requesting this endpoint often, rather only once when a dapp syncs for the first time. It will then keep this information in a local database anyway. And only requests the last x tx.

Remote nodes would have to rate limit their requests anyway somehow.

[m888m]

I second the need for transaction listings.
Regarding implementation, to compromise between system load and user friendliness, I suggest to use nonce as single argument, and implement a filter on the gui where users can limit time, counterparties, amounts etc.

[obscuren]

@frozeman that's quite an assumption you are making there and neither solves the issue at hand I described above.

@m888m the nonce is hardly the issue

[obscuren]

Remote nodes would have to rate limit their requests anyway somehow.

It's not about rate limiting either. Please reread

[rfikki]

list of transaction hashes for a given external or contract account. +1

[sammy007]

+1, with nonce param.

[m888m]

@obscuren nonce was not my main point, the need for user friendly transaction listings was. To push end user adoption, we definitely need a user-friendly way to show past transactions that happened during a certain time period and with certain counter parties. Furthermore, the transaction fees occurred for a set of transactions must be easily established.

[obscuren]

@m888m mine neither. I understand what everyone has reiterated over and over again but as I've mentioned above in great detail; there are issues with the requested feature. It's not as simple as it seems.

[tymat]

Indexing of this can be an optional setting when starting up the node. It doesn't have to be a requirement so clients that do not have CPU power can just turn the feature off.

Right now most of the web wallets and mist based wallets are forced to use some external persistence mechanism to have this data available to the user so any efforts to save CPU/etc... becomes a zero sum effort and you add an additional TCP/socket round-trip just to get the data from the blocks and then another connection to the persistence layer. Is the cost of doing indexing on the node greater than that? Probably not.

[wildbunny]

@karalabe This is a genuine question

[obscuren]

@wildbunny yes, even if your HD blows up. They are stored in the receipts of the blockchain. They are part of the consensus rules.

[obscuren]

That's why it's costing you Gas.

[wildbunny]

@obscuren Perfect, thank you

[frozeman]

@wildbunny there is a fromBlock: 0 option, which gives you passed logs you may have missed ;) Please read the RPC docs or web3.js docs

[Gustav-Simonsson]

@altsheets As I also enjoy abusing Ethereum as a currency, I started on a quick & dirty impl here: https://github.com/Gustav-Simonsson/go-ethereum/tree/list_transactions

It uses sqlite to index txs on both sender and recipient. It's WIP and pretty hacky - have to rewrite it quite a bit before it'll be ready for others to use.

Preliminary benchmarks:

• sqlite DB size after indexing all mainnet txs: 600MB
• JSON-RPC call listTransactions on address with 60 txs: <1ms. On address with 25k txs: <2s.
• Indexing from scratch: 12 hours on a ivy bridge i7 machine.

All of these metrics can probably be improved a lot, as the current hack is just a first stab and not optimised.

TODOs:

• Index during normal sync instead of traversing blockchain from HEAD to genesis.
• Handle forks & reorgs (by integrating into the code that already handles this for the leveldb instance used for blocks)
• Include txs from tx pool in API return
• Return more than the tx hash, perhaps configurable in the API call
• cmd line flags to make the call & required indexing optional, and perhaps also configure from which block to begin indexing (to save time if one knows how far back the txs one is interested in go)

[altsheets]

wow @Gustav-Simonsson you are a star!

I gather it is mainly in eth/addr_txs.go ?

Indexing from scratch: 12 hours

Oops. But 12 * 3600 / 1018621 = 0.04241 is not too bad, actually (there is the 42 again! :-) )

not optimised

Which are the bottlenecks?

Could multithreading help? Query 500 blocks at a time? This supersimple question is not time-ordered.

todo: Handle forks & reorgs

yes, and that is actually one reason why I had refrained from just doing it myself.

Fantastic work!

Rawr

[fairglu]

This kind of information is also required for 3rd party auditing, currently ethereum is hard and error-prone to audit, even for basic transfers and coin generation. There is no point in having sophisticated gas and difficulty mechanics if the monetary part is not sound and auditable (not just for isolated accounts, but for all blockchain accounts and transactions). This is also needs to be auditable efficiently and in real-time, so that fool play can be detected and acted upon quickly.

While the core ethereum team is out there to build something, and spurious transactions are more likely to be bugs than scams, this cannot be said or assumed for clones and private chains. Since the code is very complex, it is very hard to justify code and consensus can be trusted, even with multiple implementations (actually multiple implementations for a private chain will only make the 3rd party auditing more expensive).

In summary, this is not a care-bear world, without standardized auditing (of which tx listing is a key part), things lean more on the trust side than on the trust-less one.

[dmxhZGp1c2hh]

+1

Does anyone thinks that not knowing when, from who and how much you received is a cool feature?
Messages like Accounts can't display incoming transactions. does not make any sense. They can track amount, you can use them to send and receive ether, but somehow you can't track this.
Think about this from average user point of view. He needs to go to 3rd party site to check who send him ether.

As I understood the only option is to create contract. So I need to pay to receive ether? Really?

[altsheets]

:-)

[obscuren]

I can only reiterate about what has already been said which makes this feature questionable and difficult to implement correct.

listTransactions will not work for light-clients. light-clients and full clients need to support the same set of functionality. Even if full nodes would index all transactions and all calls it still wouldn't be feasible since there's no way to actually verify that the transactions the node is giving you, are valid and correct. The only part which can be validated are the transaction, but since we're also dealing with contracts (which don't send transactions but invoke an EVM opcodes) we can't actually verify that value transfers that occurred during the execution of the contract are valid (a full node might be malicious).

As I understood the only option is to create contract. So I need to pay to receive ether? Really?

Pay to receive ether? Of course not. The sender is paying, not the receiver. The only part that would cost you ether is the deployment of the contract, which costs you something like 0.001 cents, if not less. This can't be an argument not to use contracts.

[obscuren]

Please note that I'm not arguing that this isn't a good/nice/cool/epic feature. I'm arguing that, while seemingly easy to implement at first sight, is actually quite difficult to do correct, if at all.

This isn't bitcoin where the only thing you have are value transfers. Our core is much more complicated that a feature like this becomes difficult to implement or impossible to implement correctly at all.

[Joter271]

@obscuren well that explains it then. last paragraph. thanks

[dmxhZGp1c2hh]

@obscuren, understood.

[altsheets]

just to clarify, this approach #1897 (comment) is futile then?

---

EDIT: and with futile I meant, that what Gustav-Simonsson seems to have implemented already ... would lead to incorrect results? (Because if not, then just make it optional, or ship it as a standalone code/extension - and everyone's happy?). Working, correct code is the best argument possible. But some of the more recent comments since then would only be understandable, if his approach is ... simply wrong. I am confused. And that is why I asked for clarification - not for restating arguments from before his post, or inventing new ones. If there is already a solution in code, to answer the OP's question - then why argue whether such a solution should exist at all, or not? Makes no sense to me. So: Is his approach only slow/mem-hungry - or is it leading to incorrect results?

[karalabe]

@altsheets Concurrent fast sync takes 11 mins on my machine, uses 2.2GB. The proposed solution needed 12h (!!), used 0.6GB extra data. Guess the feasibility ;)

On a serious note, a pruned node does not have the data you want. Isn't stored anywhere, so to get it, you need to crunch through and replay the entire Ethereum history, so fast sync is out of the question. Pruning is out of the question. It's not part of the consensus rules, so light clients are out of the question.

The only place where this can be implemented generically is for archive nodes, but those will be very rare. Average users won't have the data needed to index such transactions.

This is analogous to the real world: if I look at 5 dollars in my wallet, I don't expect the universe to tell me all the people that ever had those banknotes. No do I want a list with all the cash I've spent and received throughout my lifetime. I know how much I have now and I don't care who gave me how much when. Note, this is for cash. Now when I open a bank account and get a credit card, I expect to see all of my transaction history, but I don't get that for free, I pay my yearly maintenance fees to access that data. It's the same with Ethereum: the Ether is a plain account is cash: you know how much you have, you see the currently pending changes, but no history; whereas a contract can act as a bank account, just you get to choose all the rules of it. The more featured you wany, the more you need to pay the bank to get it, or in this case, you need to support the network with tx fees to maintain extra data on your behalf.

Just as a closing remark, the same way as it would be awesome to have automatic past tx history for plain accounts, it would be awesome to have tx data for my personal cash expenditure... But it's not something most pepole need, and it's not sometging that bothers you in real life. Simply assume that a plain account is cold hard cash, whereas a contract is a bank account working accordong to your rules.

[karalabe]

Typed it on my phone, sorry for the many typos.

[dmxhZGp1c2hh]

@altsheets, it really make sense. The problem is that if you have bitcoin background it's kinda hard to implement same features the same way. For example my users have virtual balances and to transfer some amount to balance they use their own bitcoin address. It's easy to track, because each address belongs to some user. In ethereum I don't see same ways.

So creating account for each user is not an option. OK. I can create contract that will track transactions. But this way all my users will send ether to same contract address and how to distinguish who is who? Now I need to ask users to provide some additional data. Do I ask them to provide address, so I can track them later or to provide extra data in each transaction? Well if they use some 3rd party service mby they don't know exactly from which address ether will be send or don't have permissions to add extra data in transactions.

In bitcoin (or similar altcoin) you just create new address for every event you want to track and check later for address in transactions. Super easy and flexible.

[dmxhZGp1c2hh]

@altsheets, about implementation. listtransactions in bitcoin only returns related transaction for your wallet. So you don't need to save data about all address <-> transaction relations.

What about this logic. We have array of our account addresses (let's call it addresses). When node gets new transactions, it checks if sender or receiver is in our addresses array. If yes, we put txid of transaction in some cache table. Now we can use already implemented API to get info about transactions.

[obscuren]

@altsheets as I said, I'm not arguing the usefulness of the feature I'm arguing the solutions that have either been suggested or created. If you read my response above you'll find the answer as to why this doesn't work, i.e. The results can not be guaranteed, for anything but full nodes. Since full nodes will not be dominating our network in the future makes this feature request extremely difficult if not at all possible to implement correctly (i.e. The results can never be verified).

[dmxhZGp1c2hh]

The results can not be guaranteed, for anything but full nodes.

Is that a problem? Full node will have some additional API for tracking, makes sense.

[obscuren]

That would mean some dapps won't work with chains that were created with the fast sync option.

Mist will be our primary application to serve dapps and mist always uses the fast sync and therefore it will be impossible to support such a feature.

This feature won't work with light clients either and I expect light client to eventually dominate the network mostly.

I'm not going to make a promise but we could make such a feature available under, say, admin. This means you are free to use the feature but isn't part of standard web3 and will never be either. but please note that no dapp, including the wallet dapp or mist, will ever be able to make use of this. This would purely be considered a administrative utility.

[obscuren]

For example an exchange could use this feature if it wants to be a full/archive node indefinitely.

[fairglu]

I do not see why nodes that do not perform full validation are considered in this discussion.

Clients that have to trust other nodes will be intrinsically unsafe for services and dapps, and thus unusable, especially if/when full archive nodes are rare and sybil attacks have become easy. Any use case with real world consequences will need accounting and auditing capability, which means full history for a reasonable time frame (at least weeks, years in many cases). This means running full validating nodes (with data that can be trusted through blockchain trustless mechanisms and consensus) or... rely on "trusted" payment processors and web services to run those nodes for you.

Just as for other blockchains, even a light client is vastly more resource-intensive than relying on a web service ala blockchain.info or coinbase. And if a light client cannot provide full, validated information, it would be both less practical and less safe. There would be zero incentive in running your own light client if it cannot match the information provided by explorers like ether camp's full transparency.

So the problem may not be an easy one to solve, but it is a necessary cost to be paid for a light client to be useful in a wild, hostile world.

[obscuren]

@fairglu all of the current functionality that is supported by web3 (blocks, receipts, logs, etc) can be verified by a light client. We must consider light clients in this discussion because:

1. light clients shouldn't be any less than archive nodes in terms of web3 functionality;
2. light client and nodes synced using --fast should not be considered sub-sets of web3.

Transactions that come from the call eth_listTransactions can not be verified without having the full chain at your disposal and even then it becomes incredible impractical to validate the transactions as you'd need to parse the entire chain and figure which transactions are valid and which aren't.

The problem is however solvable by using the Ethereum system as intended as I've only pointed out so many times. If you're looking for a simple value transfer network with said functionality your best option is the Bitcoin network.

In any case DApps that rely on transaction tracking are very likely doing it wrong and I'm sure there are more than enough people willing to help solve whatever issue it you're having, including myself. You're always welcome to join our gitter to discuss your issue you're having :-)

[obscuren]

Locking this issue but leaving it open.

We're considering a suitable implementation for this feature.

No ETA.