[Bugfix][Worker] Clear NPU memory between test profiling

[shen-shanshan]

What this PR does / why we need it?

Clear NPU memory between test profiling.

If we don't clear the memory this way, the CI may get OOM error between UTs.

This modification is adapted from vllm/worker/worker.py, find more details at https://github.com/vllm-project/vllm/blob/main/vllm/worker/worker.py#L184-L186.

Does this PR introduce any user-facing change?

How was this patch tested?

[MengqingCao]

Thanks for this fixing! I think we can derectly move gc collection and peak mem status reset into NPUPlatform.empty_cache()

[shen-shanshan]

Thanks for this fixing! I think we can derectly move gc collection and peak mem status reset into NPUPlatform.empty_cache()

I have assembled the 3 methods into clear_npu_memory() in platform.

[shen-shanshan]

@wangxiyuan The CI is passed, and this PR is may needed for #969.

[MengqingCao]

Thanks for this fixing! I think we can derectly move gc collection and peak mem status reset into NPUPlatform.empty_cache()

I have assembled the 3 methods into clear_npu_memory() in platform.

Why not just do it in https://github.com/vllm-project/vllm-ascend/pull/989/files#diff-efe99563375ba645b1c1befb237632c2c91f85b21d2ab2d1095eef82aecd3999L106

[shen-shanshan]

Thanks for this fixing! I think we can derectly move gc collection and peak mem status reset into NPUPlatform.empty_cache()

I have assembled the 3 methods into clear_npu_memory() in platform.

Why not just do it in https://github.com/vllm-project/vllm-ascend/pull/989/files#diff-efe99563375ba645b1c1befb237632c2c91f85b21d2ab2d1095eef82aecd3999L106

Because maybe there are some other scenarios just need empty_cache() and don't need do the other 2 methods.

[wangxiyuan]

please merge this pr and #969 into one

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[Potabk]

hope can merge, sleep mode v1 need this

[shen-shanshan]

hope can merge, sleep mode v1 need this

I will rebase soon.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[momo609]

What is the difference between the two memory clear methods?

[shen-shanshan]

@momo609 clear_npu_memory() = gc.collect() + empty_cache() + reset_peak_memory_stats() (which will make the profile in dummy run more accurate).

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[shen-shanshan]

CC: @wangxiyuan

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[codecov]

Codecov Report

❌ Patch coverage is 40.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 27.34%. Comparing base (c30ddb8) to head (7c433f1).
⚠️ Report is 563 commits behind head on main.

Files with missing lines	Patch %	Lines
vllm_ascend/utils.py	40.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #989      +/-   ##
==========================================
- Coverage   27.39%   27.34%   -0.05%     
==========================================
  Files          56       56              
  Lines        6191     6183       -8     
==========================================
- Hits         1696     1691       -5     
+ Misses       4495     4492       -3     

Flag	Coverage Δ
unittests	27.34% <40.00%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.