Bound size of TransactionPool in memory

[birchmd]

Presently transactions are simply stored in a BTreeMap in memory, however, if we receive too many transactions too quickly, then this could grow over time, especially if the transactions are large.

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

[nikurt]

Not sure what to do if a pool reaches its maximum size but transactions keep coming in.
Is it possible that rejecting incoming transactions will also the one transaction that needs to be processed next, i.e. a transaction with the lowest nonce?
@mzhangmzz do you have suggestions?

[nikurt]

To get an idea of a reasonable tx pool size bound, will add a metric.

[bowenwang1996]

Not sure what to do if a pool reaches its maximum size but transactions keep coming in.

We might need to stop accepting transactions at that point. cc @mm-near

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

[aborg-dev]

I'll pick this issue in the context of congestion work (https://github.com/near/nearcore/milestone/26, #8878).

As a first step, I plan to investigate two naive approaches:

• Introducing hard capacity limit for transaction pool (e.g. 1000 transactions)
• Introducing TTL for transactions in the pool (e.g. 1 minute)

I highly suspect that both approaches will break some tests/assumptions that the clients make. My first goal will be to understand:

• What will actually break
• How easy would it be to fix it
• How should the clients behave in the face of these limits: Provide guidelines for retrying transaction submission #8879

[aborg-dev]

We do indeed have tests that rely on putting many transactions into the pool. Example failures are:

• https://buildkite.com/nearprotocol/nearcore/builds/27048 (with size limit 20)
• https://buildkite.com/nearprotocol/nearcore/builds/27050 (with size limit 2)

Concrete test

nearcore/integration-tests/src/tests/client/benchmarks.rs

Line 25 in 055a9e2

fn benchmark_large_chunk_production_time() {

puts 20 transactions into the pool before starting to process them.

I'm adding explicit checks for the returned transaction status #8976 which would make it much easier to catch tests broken by introducing transaction pool size limits.

Retrying the failure in all those tests would likely be too burdensome, so we'll have to use some high-enough limit that will cover the majority of tests and only introduce in the retries in tests that actually exercise the congestion scenario.

[aborg-dev]

I will proceed with implementing a hard capacity limit measured in bytes. The limit will be separate for each per-shard transaction pool within the ShardedTransactionPool.
I confirmed that none of the tests break with a limit > 1000 transactions (yet to measure how many bytes that would be).

We will also start returning a new error type to the clients when we reach the transaction pool size limit, so that they can retry this error intelligently with a back-off.

[aborg-dev]

I've also explored the option of garbage collecting transactions based on their age and while doable, the existing clients (e.g. jsonrpc) will not work well with this approach and will raise the error to a higher level, likely all the way to the user (e.g. Wallet interface). This also introduces an attack vector where users can spam the system with many transactions, knowing that only a few of them will be included within the time limit and effectively block transaction pool space at a lower cost.

The remaining steps to finish this work

• Expose the size of transaction pool and number of dropped transactions as a Prometheus metric
• Estimate reasonable default limit and set it to testnet nodes
• Suggest the limit to mainnet nodes
• Write documentation about this feature

[aborg-dev]

I've filed a bug for some follow-up work to simplify transaction pool code: #9060

[aborg-dev]

The limit of 100 MB on the per-shard transaction pool on each node will be active with the next release that @marcelo-gonzalez will be shepherding.