Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: hotspot recorder #774

Merged
merged 26 commits into from
Apr 6, 2023
Merged

Conversation

MichaelLeeHZ
Copy link
Contributor

Which issue does this PR close?

Closes #

Rationale for this change

When CeresDB becomes overloaded, users may need to identify which table is contributing the most load. In this scenario, It is useful to log hotspot write requests or read requests.

What changes are included in this PR?

Add log statistics of hotspot write requests or read requests.

Are there any user-facing changes?

None

How does this change test

By unit tests

@chunshao90 chunshao90 changed the title Feat: hotspot recorder feat: hotspot recorder Mar 27, 2023
server/src/grpc/hotspot.rs Outdated Show resolved Hide resolved
server/src/grpc/hotspot.rs Outdated Show resolved Hide resolved
server/src/limiter.rs Outdated Show resolved Hide resolved
# Conflicts:
#	server/src/config.rs
#	server/src/grpc/mod.rs
#	server/src/grpc/storage_service/mod.rs
#	server/src/limiter.rs
#	server/src/server.rs
@codecov-commenter
Copy link

codecov-commenter commented Mar 28, 2023

Codecov Report

Merging #774 (99590e0) into main (0831a84) will increase coverage by 0.14%.
The diff coverage is 78.43%.

❗ Current head 99590e0 differs from pull request most recent head 01b08f5. Consider uploading reports for the commit 01b08f5 to get more accurate results

@@            Coverage Diff             @@
##             main     #774      +/-   ##
==========================================
+ Coverage   68.18%   68.32%   +0.14%     
==========================================
  Files         295      299       +4     
  Lines       46880    47351     +471     
==========================================
+ Hits        31963    32353     +390     
- Misses      14917    14998      +81     
Impacted Files Coverage Δ
analytic_engine/src/instance/open.rs 80.69% <ø> (-0.06%) ⬇️
analytic_engine/src/manifest/details.rs 95.63% <ø> (-0.03%) ⬇️
catalog/src/schema.rs 78.26% <ø> (-0.91%) ⬇️
catalog_impls/src/table_based.rs 83.43% <ø> (-0.03%) ⬇️
common_util/src/lib.rs 100.00% <ø> (ø)
...nterpreters/src/table_manipulator/catalog_based.rs 98.13% <ø> (-0.02%) ⬇️
server/src/config.rs 36.36% <0.00%> (-0.42%) ⬇️
server/src/grpc/meta_event_service/mod.rs 0.00% <ø> (ø)
server/src/grpc/mod.rs 0.00% <0.00%> (ø)
server/src/grpc/storage_service/mod.rs 0.00% <0.00%> (ø)
... and 12 more

... and 5 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@MichaelLeeHZ MichaelLeeHZ marked this pull request as ready for review March 28, 2023 10:06
server/Cargo.toml Outdated Show resolved Hide resolved
server/src/grpc/mod.rs Outdated Show resolved Hide resolved
server/src/proxy/grpc/write.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot_lru.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/grpc/mod.rs Outdated Show resolved Hide resolved
Co-authored-by: chunshao.rcs <worcsrcsgg@163.com>
server/src/config.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
MichaelLeeHZ and others added 3 commits March 31, 2023 16:52
Co-authored-by: WEI Xikai <ShiKaiWi@users.noreply.github.com>
server/src/proxy/hotspot.rs Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
Cargo.toml Outdated Show resolved Hide resolved
server/src/grpc/mod.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Outdated Show resolved Hide resolved
server/src/proxy/hotspot.rs Show resolved Hide resolved
@ShiKaiWi ShiKaiWi enabled auto-merge April 6, 2023 09:49
Copy link
Member

@ShiKaiWi ShiKaiWi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShiKaiWi ShiKaiWi added this pull request to the merge queue Apr 6, 2023
Merged via the queue into apache:main with commit d9b7d15 Apr 6, 2023
chunshao90 added a commit to chunshao90/ceresdb that referenced this pull request May 15, 2023
* block all query

* warnings

* unit test && resolve clippy

* integration test

* resove merge

* chore

* add write fields stat

* comment

* toml sort dependency

* refact

* use clru

* resoolve test

* chore

* license

* Update server/src/proxy/hotspot.rs

Co-authored-by: chunshao.rcs <worcsrcsgg@163.com>

* chore

* Update server/src/proxy/hotspot.rs

Co-authored-by: WEI Xikai <ShiKaiWi@users.noreply.github.com>

* rm verbose comment

* chore

* chore

* fmt

* test

* fix

* chore

---------

Co-authored-by: chunshao.rcs <worcsrcsgg@163.com>
Co-authored-by: WEI Xikai <ShiKaiWi@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants