OnnxRuntime Error : sequential_executor.cc:318 Execute] Non-zero status code returned while running Transpose node. && GPU Usage Issue

[sniperHJJ]

**Error Message **
2021-07-09 02:08:35.365931341 [E:onnxruntime:, sequential_executor.cc:318 Execute] Non-zero status code returned while running Transpose node. Name:'StatefulPartitionedCall/EfficientDet-D0/bifpn/node_05/1_dn_lvl_3/input_0_up_lvl_3/1x1_pre_sample/batchnorm/FusedBatchNormV3__1010' Status Message: CUDA error cudaErrorInvalidConfiguration:invalid configuration argument
Traceback (most recent call last):
File "tfmodelTest.py", line 13, in
results_ort = sess.run(["num_detections", "detection_boxes", "detection_classes","detection_scores"], {"input_tensor": input1})
File "/usr/local/lib/python3.6/dist-packages/onnxruntime/capi/session.py", line 123, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Transpose node. Name:'StatefulPartitionedCall/EfficientDet-D0/bifpn/node_05/1_dn_lvl_3/input_0_up_lvl_3/1x1_pre_sample/batchnorm/FusedBatchNormV3__1010' Status Message: CUDA error cudaErrorInvalidConfiguration:invalid configuration argument

1. Used model : https://tfhub.dev/tensorflow/efficientdet/d0/1

2. Used Command : python -m tf2onnx.convert --saved-model ./ --output model.onnx

3. Environment
   Cython 0.29.23
   numpy 1.19.1
   onnx 1.7.0
   onnxruntime-gpu 1.5.2
   pycuda 2019.1.2
   tensorflow-gpu 2.2.0
   tf2onnx 1.10.0
   torch 1.7.1
   torch2trt 0.2.0

4. Test Code
   import onnxruntime as ort

model_path = efficientdet_d0_v12.onnx
input1 = np.ones((1, 512, 512, 3), np.uint8)

#sess = ort.InferenceSession(model_path, providers=["TensorrtExecutionProvider"])
#sess = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
sess = ort.InferenceSession(model_path, providers=["CUDAExecutionProvider"])

results_ort = sess.run(["num_detections", "detection_boxes", "detection_classes","detection_scores"], {"input_tensor": input1})

1. sess = ort.InferenceSession(model_path, providers=["TensorrtExecutionProvider"])

2. #sess = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
   Neither of the above causes an error, but it seems that the GPU is not used at all.
   I monitored it with nvidia-smi, but the GPU usage is not increasing at all.

3. sess = ort.InferenceSession(model_path, providers=["CUDAExecutionProvider"])
   "CUDAExecutionProvider" increases GPU usage, but results in the error mentioned at the beginning.

I did only the basics, but an error occurred and I can't figure it out.
Attached is the onnx file.

Any help would be appreciated.
efficientdet_d0_v12.zip

[hariharans29]

Thanks for reporting.

This was already reported (#7316) and a bug fix was shipped in 1.8.

[TomWildenhain-Microsoft]

Thanks @hariharans29, sorry should have probably suggested updating ORT first.

[zll0000]

I run a model with precision float16, some test samples can get rigth result, but some are error as follows
self._sess.run_with_iobinding(iobinding._iobinding, run_options)
RuntimeError: Error in execution: Non-zero status code returned while running Transpose node. Name:'Transpose_286' Status Message: CUDA error cudaErrorInvalidConfiguration:invalid configuration argument
however ,all the test samples can output right result by the model with precision float32
onnxruntime-gpu =1.13.1
@hariharans29