[Log] Optimize kv cache memory log from Bytes to GiB #25204
Conversation
Signed-off-by: yewentao256 <zhyanwentao@126.com>
yewentao256
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
njhill,
robertgshaw2-redhat and
ywang96
as code owners
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request aims to improve the readability of a log message by converting memory values from bytes to GiB. While the intention is good, the change introduces an issue where the suggested --kv-cache-memory command-line argument values are no longer valid for copy-pasting, as they are formatted in GiB instead of bytes. My review includes a suggestion to correct this by providing the byte value for the command and the GiB value in parentheses for readability, thus preserving the utility of the log message for users.
| f"into requested memory, or `--kv-cache-memory=" | ||
| f"{kv_cache_memory_bytes_to_gpu_limit}` " | ||
| f"({GiB(kv_cache_memory_bytes_to_gpu_limit)} GiB) to fully " |
There was a problem hiding this comment.
I don't think using GiB is valid for the cli argument
There was a problem hiding this comment.
@mgoin Thanks! I have updated the code, now it would be something like --kv-cache-memory=8511484 (xxGiB)
|
Can we make this a debug log or something to actually fix how often we see it now? |
Don't fully understand " actually fix how often we see it now", this is an default info log and will be printed each time the server launches, do you mean make this a debug info? |
|
I see this log pretty much every time I launch the server so I think something is wrong. I don't think it should be common to see this. |
Signed-off-by: yewentao256 <zhyanwentao@126.com>
@mgoin Make sense to me, just converted to debug |
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2 participants
Purpose
Actual usage is 95.97 GiB for weight, 5.42 GiB for peak activation, 2.75 GiB for non-torch memory, and 1.31 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with--kv-cache-memory=58969782886to fit into requested memory, or--kv-cache-memory=77365384192to fully utilize gpu memory. Current kv cache memory in use is 60536355430 bytes.Changing to:
Actual usage is 95.97 GiB for weight, 5.68 GiB for peak activation, 2.75 GiB for non-torch memory, and 0.0 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with--kv-cache-memory=58969782886(54.9 GiB) to fit into requested memory, or--kv-cache-memory=77365384192(73.1 GiB) to fully utilize gpu memory. Current kv cache memory in use is 38.28 GiB.