GPUs operate slower than CPUs

[sniperHJJ]

@hariharans29, @TomWildenhain-Microsoft

This is a related issue to 8352. I didn't know how to reopen the issue, so I made it again.
#8352

I tested it in onnx==1.8.0 and onnxruntime-gpu==1.8.0, CUDA 11.0, cudnn 8.0.4

CUDAExecutionProvider works without errors but it works much slower than the CPU.

I monitored with nvidia-smi and confirmed that GPU usage increases when using CUDAExecutionProvider.

CPU Test result
model time : 0.28381776809692383
model time : 0.3062574863433838
model time : 0.2735309600830078
model time : 0.28269457817077637
model time : 0.2689650058746338
model time : 0.3018655776977539
model time : 0.300079345703125
model time : 0.2846550941467285
model time : 0.2832300662994385
model time : 0.27881503105163574
model time : 0.28023219108581543
model time : 0.27510547637939453
model time : 0.27993083000183105
model time : 0.2918741703033447
model time : 0.34146547317504883
model time : 0.31770992279052734
model time : 0.2876732349395752
model time : 0.2907249927520752
model time : 0.2843780517578125
model time : 0.27753663063049316

GPU Test result
model time : 2.9515023231506348
model time : 1.0645315647125244
model time : 1.0450475215911865
model time : 1.045647382736206
model time : 1.0443611145019531
model time : 1.0493566989898682
model time : 1.0513498783111572
model time : 1.0506653785705566
model time : 1.0507018566131592
model time : 1.052443027496338
model time : 1.0558373928070068
model time : 1.049104928970337
model time : 1.0525670051574707
model time : 1.0538661479949951
model time : 1.0582499504089355
model time : 1.0533485412597656
model time : 1.0548732280731201
model time : 1.0565853118896484
model time : 1.0623681545257568
model time : 1.058274269104004

[hariharans29]

Hi @JunHyunjae ,

Please see discussion in #8166. The perf fix for EfficientNet has been merged to master. If I am not mistaken, EfficientDet should be using the same backbone as EfficientNet. Please try building from master and check the perf in your environment.