Different output with same input between dynamic and static shape model

[smarttowel]

Description

I have onnx model with fixed input from files. I try to run my model with trtexec by command:

trtexec with dynamic shape

/usr/src/tensorrt/bin/trtexec --onnx=~/model.onnx --loadInputs=descriptors_0:/home/pavel/test/descr0.dat,descriptors_1:/home/pavel/test/descr1.dat --minShapes=descriptors_0:1x256x1,descriptors_1:1x256x1 --optShapes=descriptors_0:1x256x1000,descriptors_1:1x256x1000 --maxShapes=descriptors_0:1x256x1024,descriptors_1:1x256x1024 --shapes=descriptors_0:1x256x100,descriptors_1:1x256x5 --exportOutput=/home/pavel/dynamic_debug.json --saveEngine=/home/pavel/trt.engine

I receive unexpected output from dynamic_debug.json (many numbers is zero) and try to check same onnx model with static shape by command:

trtexec with static shape

/usr/src/tensorrt/bin/trtexec --onnx=~/model.onnx --loadInputs=descriptors_0:/home/pavel/test/descr0.dat,descriptors_1:/home/pavel/test/descr1.dat --shapes=descriptors_0:1x256x100,descriptors_1:1x256x5 --exportOutput=/home/pavel/static_debug.json --saveEngine=/home/pavel/trt.engine

In this case results looks good.

How can I receive same output with dynamic shape model?

Environment

TensorRT Version: 10.6.0.26

NVIDIA GPU: NVIDIA GeForce GTX 1650 Ti

NVIDIA Driver Version: 550.127.05

CUDA Version: 12.4

CUDNN Version: 9.5.1

Operating System: Kubuntu 24.04

Relevant Files

All files is here

[lix19937]

@smarttowel Can you try to other version of trt, like trt8.6/8.5, I think trt10.6 maybe has bug in cuda-x .

BTW, I has try a test on trt8.5.10, dynamic.out is similar to static.out, the abs_diff_max=0.00048.

You can run my script, and upload the dynamic_lix.json dynamic_lix.log, static_lix.json, static_lix.log.

#!/bin/bash

trtexec --onnx=./model.onnx \
--loadInputs=descriptors_0:./descr0.dat,descriptors_1:./descr1.dat \
--minShapes=descriptors_0:1x256x1,descriptors_1:1x256x1 \
--optShapes=descriptors_0:1x256x1000,descriptors_1:1x256x1000 \
--maxShapes=descriptors_0:1x256x1024,descriptors_1:1x256x1024 \
--shapes=descriptors_0:1x256x100,descriptors_1:1x256x5 \
--exportOutput=./dynamic.json \
--dumpProfile --dumpLayerInfo --separateProfileRun  \
--profilingVerbosity=detailed \
--exportLayerInfo=dynamic_lix.json \
--verbose 2>&1 |  tee dynamic_lix.log


trtexec --onnx=./model.onnx \
--loadInputs=descriptors_0:./descr0.dat,descriptors_1:./descr1.dat \
--shapes=descriptors_0:1x256x100,descriptors_1:1x256x5 \
--exportOutput=./static.json \
--dumpProfile --dumpLayerInfo --separateProfileRun  \
--profilingVerbosity=detailed \
--exportLayerInfo=static_lix.json \
--verbose 2>&1 | tee  static_lix.log

[smarttowel]

@lix19937 Thank you for reply!

Repo was updated with new log files.
In addition, I can confirm that there no issues with TensorRT 8.6 on same hardware and TensorRT 8.5 on Orin NX R35.3.1.

[kevinch-nv]

Was able to repro locally with my Ampere card as well. Looks to be an internal library issue, we've filed a bug internally to track the fix.