Skip to content

[MISC] rename interval to max_recent_requests #14285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 26, 2025
Merged

[MISC] rename interval to max_recent_requests #14285

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4bff1eb
[MISC] rename interval to max_recent_queries
andyxning Mar 5, 2025
17b4957
Merge branch 'main' into rename_interval_to_max_recent_queries
andyxning Apr 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tests/v1/core/test_kv_cache_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,7 @@ def test_metrics():
def stats(requests, queries, hits):
return PrefixCacheStats(requests=requests, queries=queries, hits=hits)

metrics = PrefixCachingMetrics(interval=5)
metrics = PrefixCachingMetrics(max_recent_requests=5)
assert metrics.hit_rate == 0.0

metrics.observe(stats(1, 20, 9))
Expand Down
12 changes: 6 additions & 6 deletions vllm/v1/core/kv_cache_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,15 @@ class BlockHashType(NamedTuple):


class PrefixCachingMetrics:
"""Metrics for prefix caching with a hit rate of the most recent N requests.
"""Metrics for prefix caching with a hit rate of the max recent N requests.

Args:
interval: The number of the most recent requests to aggregate.
max_recent_requests: The number of the max recent requests to aggregate.
Defaults to 1000.
"""

def __init__(self, interval: int = 1000):
self.interval = interval
def __init__(self, max_recent_requests: int = 1000):
self.max_recent_requests = max_recent_requests
# The current aggregated values.
self.aggregated_requests = 0
self.aggregated_query_total = 0
Expand All @@ -70,7 +70,7 @@ def observe(self, stats: PrefixCacheStats):
are being scheduled and are looking for computed blocks.

When there are more than `interval` requests, the oldest set of
requestsare removed from the metrics.
requests are removed from the metrics.

Args:
stats: The prefix cache stats.
Expand All @@ -87,7 +87,7 @@ def observe(self, stats: PrefixCacheStats):
self.aggregated_query_hit += stats.hits

# Remove the oldest stats if the number of requests exceeds.
if self.aggregated_requests > self.interval:
if self.aggregated_requests > self.max_recent_requests:
old_requests, old_queries, old_hits = self.query_queue.popleft()
self.aggregated_requests -= old_requests
self.aggregated_query_total -= old_queries
Expand Down