Layered Transaction Pool #5290

fab-10 · 2023-03-31T17:29:27Z

PR description

This implements a new transaction pool (txpool for brevity), with the main goal to better manage nonce gaps, i.e. the possibility that the list of transactions that we see for a sender could not be in order neither contiguous, that could happen just of the way there are broadcasted on the p2p network or intentionally to try to spam the txpool with non executable transactions (transactions that could not be included in a future block), so the goal is to try to keep in the pool transactions that could be selected for a future block proposal, and a the same time, without penalizing legitimate unordered transactions, that are only temporary non executable.

It is disabled by default, to enable use the option Xlayered-tx-pool=true

The main idea is to organize the txpool in an arbitrary number of layers, where each layer has specific rules and constraints that determine if a transaction belong or not to that layer and also the way transactions move across layers.

Some design choices that apply to all layers are that a transaction can only be in one layer at any time, and that layers are chained by priority, so the first layer has the transactions that are candidate for a block proposal, and the last layer basically is where transactions are dropped. Layers are meant to be added and removed in case of specific future needs.
When adding a new transaction, it is first tried on the first layer, if it is not accepted then the next one is tried and so on.
Layers could be limited by transaction number of by space, and when a layer if full, it overflow to the next one and so on, instead when some space is freed, usually when transactions are removed since confirmed in a block, transactions from the next layer are promoted until there is space.

The current implementation is based on 3 layers, plus the last one that just drop every transaction when the previous layers are full. The 3 layers are, in order:

Prioritized
Ready
Sparse

Prioritized:

This is where candidate transactions are selected for creating a new block.
Transaction are ordered by the effective priority fee, and it is limited by size, 2000 by default, to reduce the overhead of the sorting and because that number is enough to fill any block, at the current gas limit.
Does not allow nonce gaps, and the first transaction for each sender must be the next one for that sender.
Eviction is done removing the transaction with the higher nonce for the sender of the less valuable transaction, to avoid creating nonce gaps, evicted transactions go into the next layer Ready.

Ready:

Similar to the Prioritized, it does not allow nonce gaps, and the first transaction for each sender must be the next one for that sender, but it is limited by space instead of count, thus allowing many more transactions, think about this layer like a buffer for the Prioritized.
Since it is meant to keep ten to hundred of thousand of transactions, it does not have a full ordering, like the previous, but only the first transaction for each sender is ordered using a stable value that is the max fee per gas.
Eviction is the same as the Prioritized, and evicted transaction go into the next layer Sparse.

Sparse:

This is the first layer where nonce gaps are allowed and where the first transaction for a sender could not be the next expected one for that sender.
The main purpose of this layer is to act as a purgatory for temporary unordered and/or non contiguous transactions, so that they could become ready asap the missing transactions arrive, or they are eventually evicted.
It also keep the less valuable ready transactions, that are evicted from the previous layer.
It is limited by space, and eviction select the oldest transaction first, that is sent to the End Layer that just drop it.
When promoting to the prev layer Ready, only transactions that will not create nonce gaps are selected, for that we need to keep track of the nonce distance for each sender. So we can say that is ordered by nonce distance for promotion.

Changes in Besu options

When layered txpool is enabled some existing options are ignored and new options have been added, basically due to the fact that there is no more a limit in the number of transactions that the pool can contain, but the limit is the estimated amount of memory used.

Ignored options

--tx-pool-max-size
--tx-pool-limit-by-account-percentage

New options

Note: these options and their default values are subject to change while in the experimental phase.

--Xlayered-tx-pool-layer-max-capacity the max amount of memory (in bytes) that a limited by space layer can use. Applies to Ready and Sparse layers. Default 50_000_000 bytes (50MB).
--Xlayered-tx-pool-max-prioritized the maximum number of transaction that the Prioritized layer can contain. This Layer is limited by size to reduce the work needed for sorting. Default is 2000, and has been chosen since it is more than enough to fill any block at the current gas limit.
--Xlayered-tx-pool-max-future-by-sender define the max difference between the incoming transaction nonce and the current sender nonce, for a transaction to be accepted in the txpool. This means that too far in the future transactions are ignored. Default is 200.

I have run it on mainnet and compared it against the current implementation, using mitmproxy plus this custom addon to simulate a block production every other FcU, and collected the result in this document Layered Transaction Pool metrics.ods, the results are good, and on average the Layered txpool produces with 12% more transactions in less time.

A bunch of new metrics have been added, that could be used to graph totals and details about each layer, in the official Besu Full dashboard