New txpool policy doesn't really fill the txpool

[jhoenicke]

I'm running txpool statistics to see at which fee-level transactions are processed over time, see https://jochen-hoenicke.de/queue/#ETH,24h,count. For this I run with --txpool.globalslots=150000 to ensure I see most of the pending transactions.

I recently updated to geth 1.10.13 and you can see that almost no transactions are added to the txpool since then. If I understand the new logic correctly, it will never add any transaction that pays less than the base fee. This may be not intended as it means that when the base fee drops, miners do not already have the low fee transactions in their pool but now have to wait for them to be resent over the network. Also it is a bit inconsistent, since the low fee transactions are not purged when the base fee rises but no new transactions are added below the base fee even if they pay more than the transactions already in the pool.

IMHO for txpool admission it may make more sense to use the maximum fee instead of the current effective fee, since this doesn't change over time. This is also what I use in my txpool graphs.

[Kzenox7]

I have observed the same behaviour after updating geth today. It seems like only legacy transactions are entering the mempool now. In the original issue #23837 I had also suggested to use the max fee.

For reference this is the pull request that made the change #23855

[jhoenicke]

I would suggest to have two criteria for txpool admission: (1) the maximum priority fee should be greater than the gas price set in the config (1 Gwei is the default, but miners can choose a higher or lower value) and (2) the maximum total gas price should be greater than a dynamically chosen price to keep number of queued transactions to the target size. This would roughly give the pre eip-1559 behaviour except that miners ignore eip-1559 transaction that pay them too little priority fee. It would ensure that the miners always have the transactions in the pool that are most likely to be mined by them in the future.

[Kzenox7]

(1) the maximum priority fee should be greater than the gas price set in the config

From my observations this is how it used to work before the change. However having this parameter on 1 Gwei caused the txpool to be flooded with many (non useful) low gas legacy transactions (which is why the change was made).

I believe that the ideal purpose of the -pricelimit parameter should be to be able to set a floor level for the 'effective' or 'actual' gas price of a transaction to be admitted to the pool. This would match the intended pre-EIP1559 filtering behavior. I consider the 'effective' gas price to be the BaseFee + MaxPriorityFee. However since the BaseFee varies each block it perhaps makes the admission logic too complicated.

Therefore I believe that the next best option would be to test the -pricelimit against the MaxFee and to simultaneously introduce a new EIP1559 specific variable called -MinPriorityFee or -PriorityFeeLimit.
Both could have a default value of 1 Gwei. I believe that these parameters are more intuitive and provide the flexibility that is needed.

[jhoenicke]

The effective gas price level should only be chosen dynamically. If you set it to something like 65, it would work fine at the moment but it wouldn't have enqueued any transaction in September and it would have enqueued 30k transactions in April. You don't want users to continuously reconfigure their nodes. So choosing it automatically such that the number of queued transactions hits the desired size is probably best and this is how geth worked pre-1559. In June when the base fee was 4 Gwei [edit: base fee didn't exists in June, but it would have been 4 Gwei] it would be stupid for a miner to not accept a 24 Gwei transaction. Currently even accepting all 50 Gwei transactions would just bloat your txpool.

The problem with just looking at the priority fee (as the current implementation does) is that it changes faster than the transaction pool. So during a nft sale you may have a base fee peak of 1000 Gwei but only 50 transaction in total above 500 Gwei. So why shouldn't you accept a transaction that pays over 500 Gwei? It's very likely to be the most profitable transaction in the next few minutes. It's current priority fee is -500 though so using the current priority fee as a metric would not work. If geth was consistent and removed all tx that it wouldn't accept into the pool, then it would currently clear the txpool completely whenever a block that is not 200% full is produced.

So my suggestion is to use a dynamic max gas price limit that keeps the pool to the desired size (e.g. 4000 tx), and use user set pricelimit with the priority fee only to discard transactions that don't pay the desired priority fee for the miner.

[jhoenicke]

BTW the difference between "BaseFee + MaxPriorityFee" and "Max Total Fee" is at most five minutes :) After five minutes the transaction is either mined, or the base fee has grown by so much that "Max Total Fee" is reached.

[jhoenicke]

I think the old logic before #23855 does exactly what I want. Reject transaction with too little tip cap and of the remaining keep the tx with the largest max fee cap (urgent/floating list).

I really don't understand what the problem was the pull request was about to solve but for me it changed it from working to broken. The new implementation will still store the low fee transactions in the txpool, but will not include new higher paying transactions that were sent after the base fee increased.

[Kzenox7]

The problem I had after the EIP-1559 change was that my node running with the following settings
--txpool.pricelimit 16000000000 (16 Gwei) --txpool.globalslots 250000 --txpool.globalqueue 2800
was receiving no Type 2 transactions (unless MaxPriorityFee was >= 16 Gwei), even if the effective gas price would be well over 100 Gwei.
Setting the txpool.pricelimit to 1 Gwei caused my pool to be filled with 250k (legacy) transactions and made it very slow to send transactions to.

This is what I wanted to be solved. Unless I am doing something wrong or am missing something.

[jhoenicke]

@Kzenox7 You can reduce the number of global slots. 10k should be more than enough for most application; default value is 4k. With the old code it will then do what you want, namely only retain the 10k transactions with the highest max fee cap. I use 150k just to get most of the pending transactions so I know roughly how many low fee transactions exists. But 150k will give you also the 4 Gwei transactions.

[Kzenox7]

@jhoenicke
I have downgraded my geth and omitted all of the txpool parameters. I don't quite recall why I had to add them in the past but now I'm seeing the behavior that you have described. This is indeed sufficient for most applications.
I might have added the -txpool.pricelimit for performance reasons (no processing of low gas transactions).

I think my point about the -txpool.pricelimit remains though. I didn't find the parameter meaningful pre geth 1.10.13 since it seemed to test the pricelimit against Type0.GasPrice or Type2.MaxPriorityFee (which are not the same thing at all). So I'm not sure the best way is to revert it.

My original issue remains (pre & post geth 1.10.13) but then again I'm not an expert and I'm not sure if I'm viewing the pricelimit in the right way . I also don't know Go so I think this is better suited for someone with more knowledge than me to discuss further.

[jhoenicke]

#24080 fixes my issue

[fomotrader]

@jhoenicke or @Kzenox7 I'm new to running my own node and have been using the config.toml file and here are my settings

PriceLimit = 50
PriceBump = 10
AccountSlots = 64
GlobalSlots = 15000
AccountQueue = 1048
GlobalQueue = 2800
Lifetime = 10800000000000

If I have understood things correct my node should not accept txs with less then 50 gwei because I set my price limit to do that. But when I check pending txs a lot of txs with less 20 gwei are included and they are never processed. Any tips on preventing them joining my tx pool?

Also one thing I find frustrating is the lack of clear guides out there on what units the different flags the config file uses. For example is the Lifetime in seconds, ms, microseconds etc. It's been a ton of experimentation to figure this all out.

[jhoenicke]

@fomotrader: I think the price limit is in wei, not in Gwei, so if you only want to propagate super-high-priority transactions with 50 Gwei priority fee, you should set the price limit to 50000000000. If you also want to accept normally priced transactions, you should not set it below 1 Gwei, though.

There is no option to restrict the max fee, as it doesn't make sense. Why don't you want to allow 10 Gwei transactions if the base fee goes below 10 Gwei? Or do you want to change the configuration of your node every week to adapt to the base fee?

[fomotrader]

@jhoenicke for some reason I missed this notification 😄

So in essence I noticed my mempool is filled with many txs that are underpriced and over a year old. I do on chain analytics on the txs in the mempool and seeing txs like this.

https://etherscan.io/address/0x77d2b562d34fc4609df2b2ded37bbe3f707dca23

If you decode the input data you'll see the deadline is 1633519114 which is 2021-10-06T11:18:34. Which tells me that this transaction has been going from mempool to mempool for a very long time.

I don't want my mempool to have txs that are super high priority only. But I want my it to have transactions that are no older then 24 hours since first seen. Another way to phrase this would be, I want my mempool to have txs that are likely to be included in a block within 24 hours.

This mainly started happening after EIP-1599, do you have any thoughts on this.

My initial thought was to check the type of tx and do some kind of gwei check. But the more I think about it the more that seems wrong as that value can't be dynamic and will have to be changing semi regularly.