Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add small cache to execution traces #10517

Merged
merged 3 commits into from
Mar 28, 2023

Conversation

fridrik01
Copy link
Contributor

@fridrik01 fridrik01 commented Mar 20, 2023

Related Issues

#10504

Proposed Changes

This PR adds a small cache to calls to ExecutionTrace which helps improve performance for node operators like exchanges and block explorers.

If items is in cache calls to this function will be 2-3x faster.

Additional Info

I ran a local test for 30minutes where I called lotus state compute-state --json in a loop from two threads.

  • Before this cache, each query took close to 8-10 seconds to run, but with this change it goes down to under 3 seconds.
  • Cache hit rate was 82% (671/816)

NOTE: Stress testing this over an hour resulted in Lotus process residual memory taking almost 30GB, I am kind concerned that it grew this large and GC didn't kick in, so will continue to stress test this for longer.

EDIT: Ran longer stress test, memory stayed flat so should be good

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

@fridrik01 fridrik01 force-pushed the 10504-cache-execution-traces branch from bc83c4f to 3738433 Compare March 20, 2023 15:37
@fridrik01 fridrik01 marked this pull request as ready for review March 20, 2023 16:27
@fridrik01 fridrik01 requested a review from a team as a code owner March 20, 2023 16:27
chain/stmgr/stmgr.go Outdated Show resolved Hide resolved
chain/stmgr/stmgr.go Outdated Show resolved Hide resolved
chain/stmgr/execute.go Outdated Show resolved Hide resolved
@fridrik01
Copy link
Contributor Author

Updated the PR, waiting on my lotus to sync so I can test this

chain/stmgr/execute.go Outdated Show resolved Hide resolved
Copy link
Contributor

@arajasek arajasek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This PR adds a small cache to calls to ExecutionTrace which helps
 improve performance for node operators like exchanges and block
explorers.

If items is in cache calls to this function will be 2-3x faster.

Fixes: #10504
@fridrik01 fridrik01 force-pushed the 10504-cache-execution-traces branch from 877074a to 2e45f6f Compare March 28, 2023 10:43
@fridrik01 fridrik01 requested a review from raulk March 28, 2023 11:19
@fridrik01
Copy link
Contributor Author

@raulk could you take another look at this PR so I can get it landed?

Copy link
Member

@raulk raulk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but would add an env variable to adjust the cache size. (Probably not a config value, unless we want to expose with a deep knob so openly)

@@ -39,6 +40,8 @@ import (
const LookbackNoLimit = api.LookbackNoLimit
const ReceiptAmtBitwidth = 3

const execTraceCacheSize = 16
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want to make this configurable because SPs may want to disable it while exchanges may want to crank it up? We also are going with intuition for this value, so having ability to change it without a new build would help.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will make a followup PR which makes this tuneable via env variable so we can get this landed

@fridrik01 fridrik01 merged commit b4d644e into master Mar 28, 2023
@fridrik01 fridrik01 deleted the 10504-cache-execution-traces branch March 28, 2023 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants