Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better buckets for storage proof size metrics #11075

Merged
merged 2 commits into from
Apr 16, 2024

Conversation

jancionear
Copy link
Contributor

In #11000 I added some metrics for the size of receipt/chunk storage proofs, along with upper bound estimation of those sizes. The buckets chosen in the PR weren't that good, as I didn't know what sort of values to expect.
I have spent some time looking at the data and I'd like to adjust the buckets to better fit the data distribution.

  • Most receipts generate < 100KB of storage proof, so for receipt-level sizes I'd like to use a few small buckets and then some more big buckets to catch the big outliers.
  • Chunk level proofs are usually within 2-6MB, so I changed the metric to use buckets between 0 and 15MB with 500KB increases
  • The ratio for receipts was sometimes extremely high. I think it was due to small receipts (a receipt which writes little, but removes two nodes could easily shoot up to 50x ratio). Small receipts don't matter for the hard per receipt limit, so I excluded them from the metric to keep the values reasonable.

You can see how the new metrics look like here: https://nearinc.grafana.net/d/edbl9ztm5h1q8b/stateless-validation?orgId=1&var-chain_id=mainnet&var-shard_id=All&var-node_id=ci-b20a9aef-mainnet-rpc-europe-west4-01-84346caf&from=now-15m&to=now

@jancionear jancionear added the A-stateless-validation Area: stateless validation label Apr 15, 2024
@jancionear jancionear requested a review from Longarithm April 15, 2024 13:07
@jancionear jancionear requested a review from a team as a code owner April 15, 2024 13:07
Copy link

codecov bot commented Apr 15, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.11%. Comparing base (bd80b1d) to head (52581a8).
Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #11075      +/-   ##
==========================================
- Coverage   71.24%   71.11%   -0.13%     
==========================================
  Files         760      761       +1     
  Lines      152630   152865     +235     
  Branches   152630   152865     +235     
==========================================
- Hits       108739   108714      -25     
- Misses      39454    39713     +259     
- Partials     4437     4438       +1     
Flag Coverage Δ
backward-compatibility 0.24% <0.00%> (-0.01%) ⬇️
db-migration 0.24% <0.00%> (-0.01%) ⬇️
genesis-check 1.43% <0.00%> (-0.01%) ⬇️
integration-tests 36.85% <100.00%> (+0.03%) ⬆️
linux 69.59% <73.68%> (-0.13%) ⬇️
linux-nightly 70.61% <100.00%> (-0.13%) ⬇️
macos 54.28% <73.68%> (-0.10%) ⬇️
pytests 1.66% <0.00%> (-0.01%) ⬇️
sanity-checks 1.45% <0.00%> (-0.01%) ⬇️
unittests 66.79% <73.68%> (-0.13%) ⬇️
upgradability 0.29% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jancionear jancionear added this pull request to the merge queue Apr 16, 2024
Merged via the queue into near:master with commit 30c5e14 Apr 16, 2024
29 checks passed
@jancionear jancionear deleted the buckets branch April 16, 2024 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-stateless-validation Area: stateless validation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants