Add memory consumption estimation for models in profile API. #853
Conversation
Signed-off-by: Jing Zhang <jngz@amazon.com>
jngz-es
requested review from
ylwu-amzn,
Zhangxunmt,
b4sjoo,
dhrubo-os and
rbhavna
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## 2.x #853 +/- ##
============================================
- Coverage 84.81% 84.67% -0.14%
- Complexity 1628 1631 +3
============================================
Files 135 135
Lines 6079 6116 +37
Branches 596 601 +5
============================================
+ Hits 5156 5179 +23
- Misses 666 674 +8
- Partials 257 263 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
| public synchronized void setMemSizeEstimation(String modelId, MLModelFormat format, Long size) { | ||
| Long memSize = getMemSizeEstimation(format, size); | ||
| log.debug("Updating memSizeEstimation of Model {} to {}", modelId, memSize); | ||
| getExistingModelCache(modelId).setMemSizeEstimationCPU(memSize); |
There was a problem hiding this comment.
I see we set same value for CPU and GPU, does that mean the CPU and GPU memory consumption is almost the same?
There was a problem hiding this comment.
I also have the same question. Github should have a +1 icon :)
Also what if the model is trained in the CPU but performing inference in GPU. Will the memory consumption be similar like if it's trained in GPU and performing inference in GPU?
There was a problem hiding this comment.
Yes, from my experiments the cpu, gpu memory consumption is similar.
| Double scale = 1.0; | ||
| switch (format) { | ||
| case ONNX: | ||
| scale = 1.5; |
There was a problem hiding this comment.
Can we add a comment about explanation of setting up these magic numbers?
There was a problem hiding this comment.
It is a rough estimation, actually we will deprecate it by a accurate method.
Signed-off-by: Jing Zhang <jngz@amazon.com> (cherry picked from commit dd2799a)
…rch-project#853) Signed-off-by: Jing Zhang <jngz@amazon.com>
…rch-project#853) Signed-off-by: Jing Zhang <jngz@amazon.com>
Description
Add CPU/GPU memory consumption estimation for DL models in profile API. An example is as following,
{
"nodes" : {
"jJA5JA5ES1CG71Yugpc84g" : {
"models" : {
"FMtWfYcB6ZKIXgDcq6sw" : {
"model_state" : "DEPLOYED",
"predictor" : "org.opensearch.ml.engine.algorithms.text_embedding.TextEmbeddingModel@253a6c05",
"target_worker_nodes" : [
"jJA5JA5ES1CG71Yugpc84g"
],
"worker_nodes" : [
"jJA5JA5ES1CG71Yugpc84g"
],
"mem_size_estimation_cpu" : 105529143,
"mem_size_estimation_gpu" : 105529143
},
"p1hsfIcBSHStRf2jiLu6" : {
"model_state" : "DEPLOYED",
"predictor" : "org.opensearch.ml.engine.algorithms.text_embedding.TextEmbeddingModel@76d713a3",
"target_worker_nodes" : [
"jJA5JA5ES1CG71Yugpc84g"
],
"worker_nodes" : [
"jJA5JA5ES1CG71Yugpc84g"
],
"mem_size_estimation_cpu" : 148025802,
"mem_size_estimation_gpu" : 148025802
}
}
}
}
}
Issues Resolved
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.