[Metrics] Log multi-modal cache stats and fix reset

[DarkLight1337]

Purpose

This is a redo of #16478 . It is much simplified from the original PR as we have recently consolidated the code related to multi-modal cache so that we only need to log the stats from the P0 sender cache.

• Moved SHM cache implementation to WorkerWrapperBase so that there is no need to have separate logic inside UniProcExecutor and MultiprocExecutor.
• Fix a potential issue where reset_mm_cache is not run for SHM cache. (Since clear_cache in SHM receiver cache is currently a no-op, this doesn't really matter for now)
• Fix the cache inside MultiModalBudget not being cleared after engine startup
• Fix the signature of WorkerBase.execute_model to be in line with V1 code

cc @markmc

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces metrics for the multi-modal cache, including query and hit counts. The implementation involves a nice refactoring of the existing prefix caching metrics into a more generic CachingMetrics class, which is then used for both prefix and multi-modal cache statistics. The changes are well-structured and the tests have been updated to cover both text and multimodal models. I found one issue in the test suite where an assertion relies on a coincidence in test data, which I've commented on.

[chatgpt-codex-connector]

💡 Codex Review

vllm/vllm/v1/core/kv_cache_manager.py

Lines 188 to 194 in db606d9

	if self.log_stats:
	assert self.prefix_cache_stats is not None
	if request.num_preemptions > 0:
	# Previously preempted request
	self.prefix_cache_stats.preempted_requests += 1
	self.prefix_cache_stats.preempted_queries += request.num_tokens
	self.prefix_cache_stats.preempted_hits += num_new_computed_tokens

Restore preempted prefix cache fields

The refactor in PrefixCacheStats removes the preempted_requests, preempted_queries, and preempted_hits attributes, but KVCacheManager.get_computed_blocks still increments those fields when a request is preempted. Any preempted request with stats logging enabled will now raise AttributeError: 'PrefixCacheStats' object has no attribute 'preempted_requests' before the scheduler returns, breaking request handling whenever preemption occurs. Either keep these attributes on the stats object or stop writing to them.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

[DarkLight1337]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces metrics for multi-modal cache statistics and refactors the testing framework to support both text and multi-modal models, removing obsolete v0 engine logic. The changes are well-structured, particularly the generalization of caching metrics. However, I've identified a critical issue in one of the metric tests where an expected value is calculated incorrectly, which could lead to false positives in testing.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

[DarkLight1337]

I'm unable to get tests/entrypoints/openai/test_metrics.py to pass locally even on main branch, @markmc can you try it?

[DarkLight1337]

/gemini review

[chatgpt-codex-connector]

💡 Codex Review

vllm/vllm/executor/ray_distributed_executor.py

Lines 382 to 395 in 610625d

	# Initialize the actual workers inside worker wrapper.
	all_kwargs = []
	for rank, (node_id, _) in enumerate(worker_node_and_gpu_ids):
	local_rank = node_workers[node_id].index(rank)
	kwargs = dict(
	vllm_config=self.vllm_config,
	local_rank=local_rank,
	rank=rank,
	distributed_init_method=distributed_init_method,
	is_driver_worker=(not self.parallel_config)
	or (rank % self.parallel_config.tensor_parallel_size == 0),
	)
	all_kwargs.append(kwargs)
	self._run_workers("init_worker", all_kwargs)

Ray workers never receive shared_worker_lock for SHM cache

The new WorkerWrapperBase.init_worker requires a shared_worker_lock when mm_processor_cache_type='shm' and raises a ValueError if it is missing. Multiprocess and uniprocess executors were updated to pass the lock, but the ray executor still builds all_kwargs without that argument before calling _run_workers("init_worker", all_kwargs). Any model launched via RayDistributedExecutor with SHM multimodal caching will now fail during worker initialization even though it used to run (albeit without the optimization). Please include a shared_worker_lock in the kwargs for each ray worker so the cache can be created instead of crashing.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

[gemini-code-assist]

Code Review

This pull request introduces metrics for multi-modal cache statistics and refactors the shared memory cache implementation for better code structure. The changes are well-integrated with the existing metrics and logging infrastructure, and new tests have been added to validate the functionality. However, the refactoring introduces critical breaking changes to the V0 engine execution path by modifying shared components like WorkerBase without ensuring backward compatibility. These regressions need to be addressed to prevent breaking existing V0 functionality for users.

[Isotr0py]

Seems the distributed test failure is related.

[markmc]

cc @markmc

Thanks for the cc and sorry for the delay, - metrics side lgtm!