[Misc] Accurately capture the time of loading weights #14063
Conversation
waltforme
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
njhill,
robertgshaw2-redhat and
ywang96
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
waltforme
force-pushed
the
loading-weights
branch
from
8c5d7e8 to
c67a24c
Compare
Signed-off-by: Jun Duan <jun.duan.phd@outlook.com>
waltforme
force-pushed
the
loading-weights
branch
from
c67a24c to
cbe98bf
Compare
There was a problem hiding this comment.
Thanks for the PR!
I gave this PR a try with meta-llama/Llama-3.2-1B and this is what it looks like on V0.
INFO 03-01 07:56:47 [weight_utils.py:273] Time spent downloading weights for meta-llama/Llama-3.2-1B: 14.584169 seconds
INFO 03-01 07:56:47 [weight_utils.py:307] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:05<00:00, 5.31s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:05<00:00, 5.31s/it]
INFO 03-01 07:56:52 [loader.py:423] Loading weights took 5.38 seconds
INFO 03-01 07:56:53 [model_runner.py:1117] Loading model weights took 2.3185 GB and 20.256478 seconds
I left a minor comment - please take a look!
vllm/v1/worker/gpu_model_runner.py
Outdated
| time_after_load = time.perf_counter() | ||
| self.model_memory_usage = m.consumed_memory | ||
| logger.info("Loading model weights took %.4f GB and %.6f seconds", | ||
| logger.info("Loading model took %.4f GB and %.6f seconds", |
There was a problem hiding this comment.
Let's update this on V0 as well?
I also think logger.info("Model loading took %.4f GB and %.6f seconds", sounds more natural and less confusing!
There was a problem hiding this comment.
Good point, thanks!
Latest push tweaks the wording as suggested, and contains the same change for V0.
Signed-off-by: Jun Duan <jun.duan.phd@outlook.com>
…4063) Signed-off-by: Jun Duan <jun.duan.phd@outlook.com>
…4063) Signed-off-by: Jun Duan <jun.duan.phd@outlook.com> Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>
…4063) Signed-off-by: Jun Duan <jun.duan.phd@outlook.com>
2 participants
This PR tries to better capture the time of loading weights.
As we know, there are two major phases when loading a model: (A) downloading model if not in cache, and (B) loading weights.
Due to the lazy nature of generators that
yieldtensors one by one, the downloading phase is not executed until aload_weightsmethod of the corresponding model begins to iterate over the generator.So, in order to accurately capture the time of loading weights, we need to start our stopwatch after downloading is finished and before the first tensor is
yielded. That's the general idea of this PR.