Large GPU memory usage with EXHAUSTIVE cuDNN search #7612
Labels
ep:CUDA
issues related to the CUDA execution provider
Comments
|
@duli2012 - any thought on the memory issue? |
|
I have just tested the (just released) 1.8.0 version, and the problem persists. Here's an updated table from the original issue:
|
|
Thankfully, I saw #7284 added the ability to let the memory arena shrink after each call to Ort::RunOptions options;
int deviceID = 0;
cudaGetDevice(&deviceID);
std::ostringstream stream;
stream << "gpu:" << deviceID;
options.AddConfigEntry("memory.enable_memory_arena_shrinkage", stream.str().c_str());
session.Run(options, ...);... and that worked. All the memory used by
With these numbers, we are still better off using 1.7.2. Ideally the root cause of the excessive memory usage would be solved, so we can benefit from the best performance with a reasonable memory usage. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Summary: On master with
EXHAUSTIVEcuDNN search, our model uses 5GB of GPU memory, vs only 1.3GB memory with other setups (including in TensorFlow). This memory usage cannot be reduced usinggpu_mem_limit, even though the model can actually run if there is only 0.5GB of GPU memory available.Details: We have a model converted from TensorFlow, the same model as in #7578 (in all that follows, the fix for upsampling performance mentioned in that other issue was applied to onnxruntime). Here are some statistics I collected when running this model on various versions of onnxruntime (1.7.2, or master on commit 2f04797), with different cuDNN search settings, and compared against TensorFlow:
The master branch with
EXHAUSTIVEcuDNN search has clearly the best runtime performance of all combinations, and it is the only onnxruntime setup that runs faster than Tensorflow (great job on that!). Unfortunately, it also has a dramatically larger memory usage. This is wrecking havoc with our product, which needs that memory for other purposes (and we need to keep the onnxruntime session alive to avoid the cost of re-creating it next time it is needed). In all cases I set the arena extend strategy tokSameAsRequested.I thought I could control this with the CUDA session settings. If I set
gpu_mem_limitto any value between 4GB and 8GB, the session runs but keeps using 5GB of memory. If I setgpu_mem_limitto anything less than 4GB, the session refuses to run with the following message:I then tried to artificially reduce the amount of available GPU memory by pre-allocating a big array with CUDA, and leave the memory limit to the maximum. Surprisingly, the session was then able to run, with unchanged performance:
This table suggests that the model should be able to run with
gpu_mem_limitset as low as ~1GB, yet it is not the case.nvprofreports:onnx master exhaustive
onnx master default
onnx 1.7.2 exhaustive
onnx 1.7.2 default
Tensorflow
Urgency
None as we can run with 1.7.2. Unfortunately that means we cannot benefit from the improved performance from master, which may land in the next version with this memory problem.
System information
To Reproduce
Expected behavior
Memory used should be limited to what is truly needed.
gpu_mem_limitshould behave the same as truly reducing the available GPU memory.Screenshots
N/A
Additional context
None
The text was updated successfully, but these errors were encountered: