Code

Enhance Mempool performance#226

Code

Merged

michaelsutton merged 103 commits intokaspanet:masterkaspanet/rusty-kaspa:masterfrom tiram88:mempool-perftiram88/rusty-kaspa:mempool-perfCopy head branch name to clipboardOct 6, 2023

+3,006-728

Commits

Commits on Jul 16, 2023

• Split mempool atomic validate and insert transaction in 3 steps

  tiram88

  committed
  5981bd5

  Copy full SHA for 5981bd5

• Merge branch 'master' into mempool-perf

  tiram88

  committed
  846bd38

  Copy full SHA for 846bd38

Commits on Jul 17, 2023

• Process tx relay flow received txs in batch

  tiram88

  committed
  1b18e11

  Copy full SHA for 1b18e11

Commits on Jul 18, 2023

• Use a single blocking task per MiningManagerProxy fn

  tiram88

  committed
  84c1743

  Copy full SHA for 84c1743

• Merge branch 'master' into mempool-perf

  tiram88

  committed
  a2c8e5b

  Copy full SHA for a2c8e5b

• Split parallel txs validation in chunks of max block mass

  tiram88

  committed
  563baaf

  Copy full SHA for 563baaf

Commits on Jul 20, 2023

• Merge branch 'master' into mempool-perf

  tiram88

  committed
  d842040

  Copy full SHA for d842040

• Merge branch 'master' into mempool-perf

  tiram88

  committed
  4f3d1de

  Copy full SHA for 4f3d1de

Commits on Jul 26, 2023

• Abstract expire_low_priority_transactions into Pool trait

  tiram88

  committed
  49d74eb

  Copy full SHA for 49d74eb

Commits on Aug 12, 2023

• Merge branch 'master' into mempool-perf

  tiram88

  committed
  4b36529

  Copy full SHA for 4b36529

Commits on Aug 25, 2023

• Merge branch 'master' into mempool-perf

  tiram88

  committed
  2b4d41d

  Copy full SHA for 2b4d41d

Commits on Aug 26, 2023

• Making room in the mempool for a new transaction won't remove chained txs nor parent txs of the new transaction

  tiram88

  committed
  0cb6619

  Copy full SHA for 0cb6619

Commits on Aug 29, 2023

• Refine lock granularity on Mempool and Consensus while processing unorphaned transactions (wip)

  tiram88

  committed
  6ebde53

  Copy full SHA for 6ebde53

• Fix failing test

  tiram88

  committed
  ff59ac9

  Copy full SHA for ff59ac9

Commits on Aug 30, 2023

• Enhance performance & refine lock granularity on Mempool and Consensus while revalidating high priority transactions

  tiram88

  committed
  974a980

  Copy full SHA for 974a980

• Merge branch 'master' into mempool-perf

  tiram88

  committed
  b725080

  Copy full SHA for b725080

Commits on Aug 31, 2023

• Comments

  tiram88

  committed
  b814876

  Copy full SHA for b814876

Commits on Sep 1, 2023

• Merge branch 'master' into mempool-perf

  tiram88

  committed
  68e341d

  Copy full SHA for 68e341d

• Fix upper bound of transactions chunk

  tiram88

  committed
  bd13f22

  Copy full SHA for bd13f22

• Ensure a chunk has at least 1 tx

  tiram88

  committed
  eab8c8e

  Copy full SHA for eab8c8e

• Prevent add twice the same tx to the mempool

  tiram88

  committed
  f7b58d2

  Copy full SHA for f7b58d2

• Clear transaction entries before revalidation

  tiram88

  committed
  13b4e43

  Copy full SHA for 13b4e43

• Add some logs and comments

  tiram88

  committed
  9026043

  Copy full SHA for 9026043

Commits on Sep 5, 2023

• Add logs to debug transactions removals

  tiram88

  committed
  82cb48a

  Copy full SHA for 82cb48a

• On accepted block do not remove orphan tx redeemers

  tiram88

  committed
  77af21d

  Copy full SHA for 77af21d

• Add 2 TODOs

  tiram88

  committed
  41397ba

  Copy full SHA for 41397ba

Commits on Sep 6, 2023

• Fix a bug of high priority transactions being unexpectedly orphaned or rejected

  tiram88

  committed
  54e8b39

  Copy full SHA for 54e8b39

• Refactor transaction removal reason into an enum

  tiram88

  committed
  f6225aa

  Copy full SHA for f6225aa

Commits on Sep 7, 2023

• Add an accepted transaction ids cache to the mempool and use it to prevent reentrance in mempool, broadcasting to and asking from peers

  tiram88

  committed
  ece7e97

  Copy full SHA for ece7e97

• Improve the filtering of unknown transactions in tx relay

  tiram88

  committed
  ea17aaa

  Copy full SHA for ea17aaa

• Enhance tx removal logging

  tiram88

  committed
  46fb8df

  Copy full SHA for 46fb8df

• Merge branch 'master' into mempool-perf

  tiram88

  committed
  5b5ef5d

  Copy full SHA for 5b5ef5d

• Add mempool stats

  tiram88

  committed
  8522ced

  Copy full SHA for 8522ced

Commits on Sep 10, 2023

• Process new and unorphaned blocks in topological order

  tiram88

  committed
  61ca786

  Copy full SHA for 61ca786

• Run revalidation of HP txs in a dedicated task

  tiram88

  committed
  5ba09f5

  Copy full SHA for 5ba09f5

• Some profiling and debug logs

  tiram88

  committed
  2df3029

  Copy full SHA for 2df3029

• Run expiration of LP txs in a dedicated task

  tiram88

  committed
  254a7e8

  Copy full SHA for 254a7e8

Commits on Sep 11, 2023

• remove some stopwatch calls which were timing locks

  michaelsutton

  committed
  375f78d

  Copy full SHA for 375f78d

• crucial: fix exploding complexity of handle_new_block_transactions/remove_transaction

  michaelsutton

  committed
  004beb6

  Copy full SHA for 004beb6

• fixes in on_new_block

  michaelsutton

  committed
  f8d4401

  Copy full SHA for f8d4401

• refactor block template cache into Inner

  michaelsutton

  committed
  c32b660

  Copy full SHA for c32b660

• make block_template_cache a non-blocking call (never blocks)

  michaelsutton

  committed
  625cf12

  Copy full SHA for 625cf12

Commits on Sep 12, 2023

• Log build_block_template retries

  tiram88

  committed
  205e934

  Copy full SHA for 205e934

• While revalidating HP txs, only recheck transaction entries

  tiram88

  committed
  674481f

  Copy full SHA for 674481f

• Fix accepted count during revalidation

  tiram88

  committed
  8420832

  Copy full SHA for 8420832

Commits on Sep 13, 2023

• mempool bmk: use client pools + various improvements

  michaelsutton

  committed
  78fefb3

  Copy full SHA for 78fefb3

• Merge branch 'mempool-perf' of https://github.com/tiram88/rusty-kaspa into mempool-perf

  tiram88

  committed
  5bd630c

  Copy full SHA for 5bd630c

• Improve the topological sorting of transactions

  tiram88

  committed
  118559e

  Copy full SHA for 118559e

Commits on Sep 14, 2023

• Return transaction descendants BFS ordered + some optimizations

  tiram88

  committed
  9302574

  Copy full SHA for 9302574

• Group expiration and revalidation of mempool txs in one task

  tiram88

  committed
  b63b586

  Copy full SHA for b63b586

• Refine the schedule of the cleaning task

  tiram88

  committed
  55273ab

  Copy full SHA for 55273ab

Commits on Sep 15, 2023

• ignore perf logs

  michaelsutton

  committed
  a9a93a0

  Copy full SHA for a9a93a0

• maintain mempool ready transactions in a dedicated set

  michaelsutton

  committed
  782315f

  Copy full SHA for 782315f

• Bound the returned candidate transactions to a maximum

  tiram88

  committed
  0614dcb

  Copy full SHA for 0614dcb

• Merge pull request #4 from michaelsutton/mempool-perf-2

  Show description for 66b2ea0
  tiram88

  authored
  66b2ea0

  Copy full SHA for 66b2ea0

Commits on Sep 17, 2023

• Reduces the max execution time of build block template

  tiram88

  committed
  21e3d1c

  Copy full SHA for 21e3d1c

• lint

  tiram88

  committed
  9c0acba

  Copy full SHA for 9c0acba

• Add mempool lock granularity to get_all_transactions

  tiram88

  committed
  e320045

  Copy full SHA for e320045

• Restore block template cache lifetime & make it customizable in devnet-prealloc feature

  tiram88

  committed
  84b0476

  Copy full SHA for 84b0476

• Restore block template cache lifetime & make it customizable in devnet-prealloc feature

  tiram88

  committed
  8212a05

  Copy full SHA for 8212a05

• Merge branch 'mempool-perf' of https://github.com/tiram88/rusty-kaspa into mempool-perf

  tiram88

  committed
  403c6b0

  Copy full SHA for 403c6b0

Commits on Sep 18, 2023

• Relax a bit the BBT maximum attempts constraint

  tiram88

  committed
  01c8f9c

  Copy full SHA for 01c8f9c

Commits on Sep 19, 2023

• Refactor multiple contained_by_txs fns into one generic

  tiram88

  committed
  d024904

  Copy full SHA for d024904

• Test selector transaction rejects & fix empty template returned by select_transactions upon selector reuse

  tiram88

  committed
  1f0c44e

  Copy full SHA for 1f0c44e

• Log some mempool metrics

  tiram88

  committed
  bb56c70

  Copy full SHA for bb56c70

• Merge branch 'master' into mempool-perf

  tiram88

  committed
  1156ee7

  Copy full SHA for 1156ee7

Commits on Sep 20, 2023

• Handle new block and then new block template

  tiram88

  committed
  14da8a2

  Copy full SHA for 14da8a2

Commits on Sep 21, 2023

• turn tx selector into an ongoing process with persistent state (wip: some tests are broken; selector is not used correctly by builder)

  michaelsutton

  committed
  92bb8dd

  Copy full SHA for 92bb8dd

• use tx selector for BBT (wip: virtual processor retry logic)

  michaelsutton

  committed
  0e54b66

  Copy full SHA for 0e54b66

• virtual processor selector retry logic

  michaelsutton

  committed
  d231134

  Copy full SHA for d231134

• make BBT fallible by some selector criteria + comments and some docs

  michaelsutton

  committed
  9a015d1

  Copy full SHA for 9a015d1

• add an infallible mode to virtual processor build_block_template()

  michaelsutton

  committed
  58e0ed5

  Copy full SHA for 58e0ed5

• constants for tx selector successful decision

  michaelsutton

  committed
  827b9fa

  Copy full SHA for 827b9fa

• Add e-tps to logged mempool metrics

  tiram88

  committed
  db0cddf

  Copy full SHA for db0cddf

• avoid realloc

  michaelsutton

  committed
  08456c6

  Copy full SHA for 08456c6

Commits on Sep 22, 2023

• Address review comments

  tiram88

  committed
  099497b

  Copy full SHA for 099497b

• Merge pull request #5 from michaelsutton/mempool-perf-3

  Show description for 70a91ec
  tiram88

  authored
  70a91ec

  Copy full SHA for 70a91ec

• Use number of ready txs in e-tps & enhance mempool lock

  tiram88

  committed
  bb724e6

  Copy full SHA for bb724e6

• Ignore failing send for clean tokio shutdown

  tiram88

  committed
  66dbca8

  Copy full SHA for 66dbca8

Commits on Sep 23, 2023

• Log double spends

  tiram88

  committed
  abd3143

  Copy full SHA for abd3143

Commits on Sep 24, 2023

• Log tx script cache stats (wip)

  tiram88

  committed
  c29680f

  Copy full SHA for c29680f

• Ease atomic lock ordering & enhance counter updates

  tiram88

  committed
  6f8b024

  Copy full SHA for 6f8b024

Commits on Sep 26, 2023

• Enhance tx throughput stats log line

  tiram88

  committed
  87f20f0

  Copy full SHA for 87f20f0

• More robust management of cached data life cycle

  tiram88

  committed
  bc6bdc6

  Copy full SHA for bc6bdc6

• Log mempool sampled instead of exact lengths

  tiram88

  committed
  1c7357d

  Copy full SHA for 1c7357d

Commits on Oct 3, 2023

• avoid passing consensus to orphan pool

  michaelsutton

  committed
  fd4731e

  Copy full SHA for fd4731e

• rename ro validate_transaction_unacceptance and move to before the orphan case (accepted txs will usually be orphan)

  michaelsutton

  committed
  8bec918

  Copy full SHA for 8bec918

Commits on Oct 5, 2023

• rename cleaning -> mempool_scanning

  michaelsutton

  committed
  a99d3d8

  Copy full SHA for a99d3d8

• keep intervals aligned using a round-up formula (rather than a loop)

  michaelsutton

  committed
  f161b1c

  Copy full SHA for f161b1c

• design fix: avoid exposing full collections as mut. This violates encapsulation logic since collections can be completely modified externally; while in tx pools it is important to make sure various…

  Show description for 41e4a94
  michaelsutton

  committed
  41e4a94

  Copy full SHA for 41e4a94

• minor: close all pool receivers on op error

  michaelsutton

  committed
  316f9d1

  Copy full SHA for 316f9d1

• remove_transaction: no need to manually update parent-child relations in the case remove_redeemers=false. This is already done via remove_transaction_from_sets -> `transaction_pool.remove_tra…

  Show description for 8ce99e7
  michaelsutton

  committed
  8ce99e7

  Copy full SHA for 8ce99e7

• encapsulate remove_transaction_utxos into transaction_pool

  michaelsutton

  committed
  4faf2fa

  Copy full SHA for 4faf2fa

• no need to remove_redeemers_of for the initial removed tx since this happens as part of:

  Show description for 7be8b61
  michaelsutton

  committed
  7be8b61

  Copy full SHA for 7be8b61

• inline remove_from_transaction_pool_and_update_orphans

  michaelsutton

  committed
  e6142e7

  Copy full SHA for e6142e7

• remove redeemers of expired low-prio txs + register scan time and daa score after collection (bug fix)

  michaelsutton

  committed
  044fa3d

  Copy full SHA for 044fa3d

• change mempool monitor logs to debug

  michaelsutton

  committed
  969fae4

  Copy full SHA for 969fae4

Commits on Oct 6, 2023

• Merge branch 'master' into mempool-perf

  michaelsutton

  committed
  d01ebd6

  Copy full SHA for d01ebd6

• Merge branch 'master' into mempool-perf

  michaelsutton

  committed
  acc3016

  Copy full SHA for acc3016

• make tps logging more accurate

  michaelsutton

  committed
  2e11cf6

  Copy full SHA for 2e11cf6

• Merge branch 'master' into mempool-perf

  michaelsutton

  committed
  cc51b62

  Copy full SHA for cc51b62

• import bmk improvements from mempool-perf-stats branch

  michaelsutton

  committed
  5aff480

  Copy full SHA for 5aff480

• make config.block_template_cache_lifetime non-feature dependent

  michaelsutton

  committed
  099aa40

  Copy full SHA for 099aa40