INT64 clamping to INT32 creates overhead while using TensorRT

[JingyaHuang]

System Info

optimum: dev
CUDA: 11.3
cuDNN: 8.3.2
TensorRT: 8.4.1.5

Who can help?

@JingyaHuang

Reproduction

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

ort_model = ORTModelForSequenceClassification.from_pretrained(
    "philschmid/tiny-bert-sst2-distilled",
    from_transformers=True,
    provider="TensorrtExecutionProvider",
)

tokenizer = AutoTokenizer.from_pretrained("philschmid/tiny-bert-sst2-distilled")
inp = tokenizer("expectations were low, actual enjoyment was high", return_tensors="pt", padding=True)

result = ort_model(**inp)
assert ort_model.providers == ["TensorrtExecutionProvider", "CUDAExecutionProvider", "CPUExecutionProvider"]

Log

2022-12-21 09:40:48.208932701 [W:onnxruntime:Default, tensorrt_execution_provider.h:60 log] [2022-12-21 09:40:48 WARNING] external/onnx-tensorrt/onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32

Expected behavior

Clamping should be done before the inference.

Will work on a convert_int64_to_int32() function to solve it.

[fxmarty]

@JingyaHuang the warning

2022-12-21 11:03:15.553857281 [W:onnxruntime:Default, tensorrt_execution_provider.h:60 log] [2022-12-21 10:03:15 WARNING] external/onnx-tensorrt/onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.

is raised at the model initialization, so I'm not sure what we can do more? The inputs themselves being int64 from the tokenizer.

[fxmarty]

Have a working prototype for this. trtexec comes very handy to test out. The warnings come due to Slice operator having end indexes as the maximum representable int64 value. Casting to np.iinfo(np.int32).max + astype(np.int32) removes the warnings.