RuntimeError when converting the pytorch model to onnx model

[Chao86]

hello, @ppwwyyxx , I’m using the API export_onnx to convert the model, however , it encounters a runtimeError. My code and runtimeerror as following：

Code

cfg = setup_cfg(args) model = build_model(cfg) checkpointer = DetectionCheckpointer(model) checkpointer.resume_or_load("/data2/User/heng_cao/model.pth") model.eval() register_dataset() data_loader = build_detection_test_loader(cfg, dataset_name=cfg.DATASETS.TEST, mapper=val_mapper) first_batch = next(iter(data_loader)) tracer = Caffe2Tracer( cfg, model, first_batch) onnx_model = tracer.export_onnx() onnx.save(onnx_model, "model.onnx")

the error as following:

(detectron2) heng_cao@amax:~/Project$ /home/heng_cao/anaconda3/envs/detectron2/bin/python /home/heng_cao/Project/pytorch2onnx/model2onnx.py /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/onnx/utils.py:255: UserWarning: add_node_names' can be set to True only when 'operator_export_type' is ONNX. Since 'operator_export_type' is not set to 'ONNX', add_node_names argument will be ignored.
warnings.warn("{}' can be set to True only when 'operator_export_type' is " /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/onnx/utils.py:255: UserWarning: do_constant_folding' can be set to True only when 'operator_export_type' is ONNX. Since 'operator_export_type' is not set to 'ONNX', do_constant_folding argument will be ignored.
warnings.warn("`{}' can be set to True only when 'operator_export_type' is "
[E context_gpu.cu:455] Encountered following in ~CUDAContext: [enforce fail at context_gpu.h:197] error == cudaSuccess. 2 vs 0. Error at: /opt/conda/conda-bld/pytorch_1623448234945/work/caffe2/core/context_gpu.h:197: out of memory
frame $#$0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x47 (0x7fda859447a7 in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: + 0x2989c97 (0x7fda8851fc97 in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #2: caffe2::CUDAContext::~CUDAContext() + 0x97 (0x7fda88520f37 in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #3: + 0x2b4e904 (0x7fda886e4904 in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x2b4ee1a (0x7fda886e4e1a in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #5: + 0x2b4dc64 (0x7fda886e3c64 in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #6: + 0x37380bf (0x7fdacf88e0bf in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #7: + 0xa96682 (0x7fdb1fe11682 in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #8: + 0xa69b6c (0x7fdb1fde4b6c in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #9: + 0xa6a3bf (0x7fdb1fde53bf in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #10: + 0x4fdd6e (0x7fdb1f878d6e in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #11: PyCFunction_Call + 0x54 (0x55d219165834 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #12: _PyObject_MakeTpCall + 0x158 (0x55d219126da8 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #13: _PyEval_EvalFrameDefault + 0x1bf6 (0x55d219182ea6 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #14: _PyEval_EvalCodeWithName + 0x886 (0x55d219153066 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #15: _PyFunction_Vectorcall + 0x1ff (0x55d21915385f in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #16: + 0x145e14 (0x55d219100e14 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #17: + 0x1a3376 (0x55d21915e376 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #18: + 0x146149 (0x55d219101149 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #19: _PyFunction_Vectorcall + 0x108 (0x55d219153768 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #20: PyVectorcall_Call + 0x6e (0x55d219127ede in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #21: _PyEval_EvalFrameDefault + 0x4e70 (0x55d219186120 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #22: _PyEval_EvalCodeWithName + 0x952 (0x55d219153132 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #23: _PyFunction_Vectorcall + 0x19b (0x55d2191537fb in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #24: + 0x1a3549 (0x55d21915e549 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #25: PyVectorcall_Call + 0x6e (0x55d219127ede in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #26: _PyEval_EvalFrameDefault + 0x4e70 (0x55d219186120 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #27: _PyEval_EvalCodeWithName + 0x1e9 (0x55d2191529c9 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #28: _PyFunction_Vectorcall + 0x19b (0x55d2191537fb in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #29: + 0x1a3549 (0x55d21915e549 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #30: PyVectorcall_Call + 0x6e (0x55d219127ede in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #31: _PyEval_EvalFrameDefault + 0x4e70 (0x55d219186120 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #32: _PyEval_EvalCodeWithName + 0x1e9 (0x55d2191529c9 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #33: _PyFunction_Vectorcall + 0x19b (0x55d2191537fb in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #34: _PyObject_FastCallDict + 0x56 (0x55d21913e1c6 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #35: _PyObject_Call_Prepend + 0x67 (0x55d2191aba27 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #36: + 0x1f0aa8 (0x55d2191abaa8 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #37: PyObject_Call + 0x61 (0x55d219128061 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #38: _PyEval_EvalFrameDefault + 0x4e70 (0x55d219186120 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #39: _PyEval_EvalCodeWithName + 0x952 (0x55d219153132 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #40: _PyFunction_Vectorcall + 0x19b (0x55d2191537fb in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #41: PyVectorcall_Call + 0x6e (0x55d219127ede in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #42: + 0xb0fe9b (0x7fdb1fe8ae9b in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #43: + 0xb54f84 (0x7fdb1fecff84 in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #44: torch::jit::tracer::trace(std::vector<c10::IValue, std::allocatorc10::IValue >, std::function<std::vector<c10::IValue, std::allocatorc10::IValue > (std::vector<c10::IValue, std::allocatorc10::IValue >)> const&, std::function<std::string (at::Tensor const&)>, bool, bool, torch::jit::Module*, std::vector<std::string, std::allocatorstd::string > const&) + 0x5ae (0x7fdacf4f661e in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #45: + 0xb50b44 (0x7fdb1fecbb44 in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #46: + 0xb56b9e (0x7fdb1fed1b9e in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #47: + 0x4fdd6e (0x7fdb1f878d6e in /home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #48: PyCFunction_Call + 0x54 (0x55d219165834 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #49: _PyObject_MakeTpCall + 0x158 (0x55d219126da8 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #50: _PyEval_EvalFrameDefault + 0x4448 (0x55d2191856f8 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #51: _PyEval_EvalCodeWithName + 0x952 (0x55d219153132 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #52: _PyFunction_Vectorcall + 0x19b (0x55d2191537fb in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #53: + 0x1a3549 (0x55d21915e549 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #54: PyVectorcall_Call + 0x6e (0x55d219127ede in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #55: _PyEval_EvalFrameDefault + 0x4e70 (0x55d219186120 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #56: _PyEval_EvalCodeWithName + 0x1e9 (0x55d2191529c9 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #57: _PyFunction_Vectorcall + 0x19b (0x55d2191537fb in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #58: _PyObject_FastCallDict + 0xd2 (0x55d21913e242 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #59: _PyObject_Call_Prepend + 0x67 (0x55d2191aba27 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #60: + 0x1f0aa8 (0x55d2191abaa8 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #61: PyObject_Call + 0x61 (0x55d219128061 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #62: _PyEval_EvalFrameDefault + 0x4e70 (0x55d219186120 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)
frame #63: _PyEval_EvalCodeWithName + 0x886 (0x55d219153066 in /home/heng_cao/anaconda3/envs/detectron2/bin/python)

Traceback (most recent call last):
File "/home/heng_cao/Project/pytorch2onnx/model2onnx.py", line 131, in
onnx_model = tracer.export_onnx()
File "/home/heng_cao/detectron2/detectron2/export/api.py", line 121, in export_onnx
return export_onnx_model_impl(self.traceable_model, (self.traceable_inputs,))
File "/home/heng_cao/detectron2/detectron2/export/caffe2_export.py", line 56, in export_onnx_model
torch.onnx.export(
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/onnx/init.py", line 275, in export
return utils.export(model, args, f, export_params, verbose, training,
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/onnx/utils.py", line 88, in export
_export(model, args, f, export_params, verbose, training, input_names, output_names,
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/onnx/utils.py", line 689, in _export
_model_to_graph(model, args, verbose, input_names,
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/onnx/utils.py", line 458, in _model_to_graph
graph, params, torch_out, module = _create_jit_graph(model, args,
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/onnx/utils.py", line 422, in _create_jit_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args)
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/onnx/utils.py", line 373, in _trace_and_get_graph_from_model
torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/jit/_trace.py", line 1160, in _get_trace_graph
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward
graph, out = torch._C._create_graph_by_tracing(
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/jit/_trace.py", line 118, in wrapper
outs.append(self.inner(*trace_inputs))
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1039, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/heng_cao/anaconda3/envs/detectron2/lib/python3.8/contextlib.py", line 75, in inner
return func(*args, **kwds)
File "/home/heng_cao/detectron2/detectron2/export/caffe2_modeling.py", line 270, in forward
images = self._caffe2_preprocess_image(inputs)
File "/home/heng_cao/detectron2/detectron2/export/caffe2_modeling.py", line 209, in _caffe2_preprocess_image
data = alias(data, "data")
File "/home/heng_cao/detectron2/detectron2/export/shared.py", line 596, in alias
return torch.ops._caffe2.AliasWithName(x, name, is_backward=is_backward)
RuntimeError: [enforce fail at context_gpu.h:197] error == cudaSuccess. 2 vs 0. Error at: /opt/conda/conda-bld/pytorch_1623448234945/work/caffe2/core/context_gpu.h:197: out of memory`

Can you offer some help to tackle this? Thanks in advance.

[github-actions]

You've chosen to report an unexpected problem or bug. Unless you already know the root cause of it, please include details about it by filling the issue template.
The following information is missing: "Instructions To Reproduce the Issue and Full Logs"; "Your Environment";

[ppwwyyxx]

If you need help to solve an unexpected issue you observed, please provide runnable code that can reproduce the issue.
Using official models & weights can help create code runnable by others.

[github-actions]

Requested information was not provided in 7 days, so we're closing this issue.

Please open new issue if information becomes available. Otherwise, use github discussions for free-form discussions.