JSON-RPC edge-cases, nonce management and eth_getTransactionBySenderAndNonce

[alcuadrado]

During the JSON-RPC meeting in Istanbul, I made a case that eth_getTransactionBySenderAndNonce would be useful for dapp and tool developers. I'm opening this issue to continue the discussion here, and to go into more detail.

Problems with the JSON-RPC

These are the problems that I think could be solved with eth_getTransactionBySenderAndNonce.

Problem 1: Losing track of a transaction

I created this pictures of how a the normal workflow of sending a transaction from a dapp or tool that interacts with a wallet looks like:

As shown above, the flow would be something like this:

1. The dapp/tool wants to send a transaction, so it calls into a library like ethers/web3/viem.
2. The library is connected to a wallet, which exposes the same JSON-RPC interface as a client. Note that the wallet doesn't just forward things to a client, it has its own implementation of some methods, like eth_sendTransaction.
3. To send a transaction, the library sends an eth_sendTransaction to the wallet.
4. The wallet receives that request and creates and signs a transaction locally.
5. At this point, the wallet is already aware of the full tx body and hence its hash, but it hasn't communicated it to the the library yet.
6. The wallet will then do an eth_sendRawTransaction, sending the full tx body to an EL client, which will broadcast it.
7. The wallet returns a hash as the response of eth_sendTransaction, which the library receives and passes to the dapp.

If after sending the tx to the EL client anything goes wrong with the wallet, library or dapp, a transaction will be broadcasted as requested by the dapp, but the dapp will never be able to access it.

Note that the order of 6 and 7 are unspecified. The wallet could return the hash before sending the raw transaction. As it's unclear, we are forced to assume the worst situation is possible.

If we had eth_getTransactionBySenderAndNonce a dapp could keep track of the nonce and transaction params, and resume from errors if needed, by checking if a transaction with that sender and nonce exists, and if it has the same params as the one that it intended to send.

Problem 2: Replaced and dropped transactions

When a dapp/tool sends a transaction, the tx can be replaced by another one with the same nonce and sender, or dropped from the mempool. What's more, replacement transactions can be equivalent to the one you sent (e.g. just higher fees), or a completely different one (e.g. to a different account).

All of these situations manifest as a eth_getTransactionByHash returning null.

The way to distinguish between some of them (replaced vs dropped) is by using eth_getTransactionCount with different blocktags and analyzing a pretty complex set of cases, including making tons of assumptions about how "pending" works in this case.

If we had eth_getTransactionBySenderAndNonce a dapp could use it to understand what happened if eth_getTransactionByHash returns null.

Proposal

Add a method eth_getTransactionBySenderAndNonce with two parameters:

1. An address: the sender of the transaction.
2. A quantity: the nonce of the transaction.

The JSON-RPC server should return the hash of a transaction (or the full tx?) of the transaction sent from the provided address using that nonce. If no such transaction exists, it should return null.

If multiple transactions from a sender and nonce were received, I assume there's one that can be considered the best candidate for inclusion (e.g. pays more fees), and that one should be used.

Things to consider

1. Does this method have other use cases?
2. Are there any alternative solution?
3. How many resources would this method require for transactions that are still in the mempool?
4. How many resources would this method require for transactions that were included in a block?
5. How many blocks are really important to solve the problems listed above?
6. How can RPC/Node providers (e.g. Infura) support this?

Tagging people present on the in-person discussion, or who I've discussed this in the past: @s1na @lightclient @sambacha @kanej

[alcuadrado]

This is also relevant: https://github.com/manifoldfinance/rpc-eip-drafts/blob/master/eth_getTransactionBySenderAndNonce.md

[alcuadrado]

I just found that it's already available in otterscan: https://github.com/otterscan/otterscan/blob/develop/docs/custom-jsonrpc.md#ots_gettransactionbysenderandnonce

[spalladino]

This issue is so commonplace that when ethers.js waits for a transaction to be mined, instead of polling using the tx hash, it downloads every single tx on every single block to try and find the tx or a replacement. Needless to say, this would be better served by this RPC call.

[spalladino]

FWIW this call is also available in Erigon

[wmitsuda]

hello, Willian from Otterscan here, I wrote the original spec and implementation in Erigon for usage in Otterscan.

Happy to rename it and make adjustments in Erigon if it’s going to be promoted to eth namespace.

I need to add that current implementation only considers txs already included in a block, i.e. no indexing of txpool

[alcuadrado]

I need to add that current implementation only considers txs already included in a block, i.e. no indexing of txpool

That's interesting. Any reason for that?

[s1na]

Awesome thanks for the thorough description!

I'd like to add more context from the IRL conversation. First point is that ELs will have to store an additional index to serve this data. But we knew this, nothing new.

The other point which @alcuadrado briefly touched on in the questions is the question of pending vs mined transactions. Right off the bat, I think RPC providers simply cannot return pending transactions. Given the heavy dependence of wallets on providers, it begs the question, is it enough to return only mined transactions by their sender and nonce? It seems they can benefit from this method, but it's not enough? the old way of checking things still has to be kept in case the tx hasn't been mined.

[antonydenyer]

The other point which @alcuadrado briefly touched on in the questions is the question of pending vs mined transactions. Right off the bat, I think RPC providers simply cannot return pending transactions. Given the heavy dependence of wallets on providers, it begs the question, is it enough to return only mined transactions by their sender and nonce? It seems they can benefit from this method, but it's not enough? the old way of checking things still has to be kept in case the tx hasn't been mined.

Indeed, as a rule of thumb, RPC providers care about the consistency of their responses to users. A node provider will typically load balance requests to different nodes and will therefore never have a consistent state for pending transactions, consequently most disable it.

[sambacha]

I need to add that current implementation only considers txs already included in a block, i.e. no indexing of txpool

That's interesting. Any reason for that?

txpool related rpc methods are generally privileged.

There exists an issue in which the same tx and nonce are replayed with higher gas price and eviction has not taken place

[sambacha]

Here is the issue for our use case, separate of @alcuadrado independent use case

manifoldfinance/mevETH2#175

[wmitsuda]

I need to add that current implementation only considers txs already included in a block, i.e. no indexing of txpool

That's interesting. Any reason for that?

no special reason, it just happened that we didn't need it.

[sambacha]

I just found that it's already available in otterscan: otterscan/otterscan@develop/docs/custom-jsonrpc.md#ots_gettransactionbysenderandnonce

ots_getTransactionBySenderAndNonce is also available via anvil now too

The other point which @alcuadrado briefly touched on in the questions is the question of pending vs mined transactions. Right off the bat, I think RPC providers simply cannot return pending transactions. Given the heavy dependence of wallets on providers, it begs the question, is it enough to return only mined transactions by their sender and nonce? It seems they can benefit from this method, but it's not enough? the old way of checking things still has to be kept in case the tx hasn't been mined.

This is a good question, and in fact there already exists such providers that sort of do return pending transaction hashes to share (e.g. flashbots mev share). There is a use case for the RPC method to be supported by block builders. If we are also going to make the distinction between pending and mined, we should consider supporting the special strings:

Block Tags	Description
finalized	The most recent crypto-economically secure block cannot be re-orged outside of manual intervention driven by community coordination
safe	The most recent block that is safe from re-orgs under honest majority and certain synchronicity assumptions
unsafe	The most recent block in the canonical chain observed by the client this block can be re-orged out of the canonical chain
pending	A sample next block built by the client on top of unsafe and containing the set of transactions usually taken from local mempool

[sambacha]

Just giving an update, this has been added as a possible idea for the latest Ethereum Protocol Cohort to work on, see: https://github.com/eth-protocol-fellows/cohort-five/blob/main/projects/project-ideas.md#:~:text=%E2%81%83%20Specification%20and%20implementation%20of%20eth_getTransactionBySenderAndNonce%2C%20see%20ethereum/execution%2Dapis%23494