Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decreasing inference time on cpu #31

Open
M-Zubair10 opened this issue Apr 9, 2023 · 6 comments
Open

Decreasing inference time on cpu #31

M-Zubair10 opened this issue Apr 9, 2023 · 6 comments

Comments

@M-Zubair10
Copy link

Thanks for this awesome model, it does evaluate good with pretrained model

Right now i am getting 15s average inference time, any way to reduce it to 2-3s

@SlongLiu
Copy link
Collaborator

It is a good question, but we have not explored it yet. The straightest way is to deploy the model like ONNX.

@GeorgePearse
Copy link
Contributor

GeorgePearse commented Apr 12, 2023

@SlongLiu should the operations in the model already support ONNX export? e.g.

code vaguely similar to the below should work?

# Export the model
torch.onnx.export](https://pytorch.org/docs/stable/onnx.html#torch.onnx.export)(torch_model,               # model being run
                  x,                         # model input (or a tuple for multiple inputs)
                  "super_resolution.onnx",   # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=10,          # the ONNX version to export the model to
                  do_constant_folding=True,  # whether to execute constant folding for optimization
                  input_names = ['input'],   # the model's input names
                  output_names = ['output'], # the model's output names
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable length axes
                                'output' : {0 : 'batch_size'}})
                                

I'll look in to optimising it with tools like OpenVINO if it does.

@Weizhongjin
Copy link

i try to use torch.onnx.export to transfer grounding-dino to onnx, it seems like some problem , such as several logical operators are not supported by onnx. for example:
torch.onnx.symbolic_registry.UnsupportedOperatorError: Exporting the operator ::_ior to ONNX opset version 13 is not supported
do you know which part of model make this problem. @GeorgePearse

@oylz
Copy link

oylz commented Jun 28, 2023

here

@Dratlan
Copy link

Dratlan commented Aug 1, 2023

i try to use torch.onnx.export to transfer grounding-dino to onnx, it seems like some problem , such as several logical operators are not supported by onnx. for example: torch.onnx.symbolic_registry.UnsupportedOperatorError: Exporting the operator ::_ior to ONNX opset version 13 is not supported do you know which part of model make this problem. @GeorgePearse

Have you solve it please

@kobic8
Copy link

kobic8 commented Aug 10, 2023

@SlongLiu should the operations in the model already support ONNX export? e.g.

code vaguely similar to the below should work?

# Export the model
torch.onnx.export](https://pytorch.org/docs/stable/onnx.html#torch.onnx.export)(torch_model,               # model being run
                  x,                         # model input (or a tuple for multiple inputs)
                  "super_resolution.onnx",   # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=10,          # the ONNX version to export the model to
                  do_constant_folding=True,  # whether to execute constant folding for optimization
                  input_names = ['input'],   # the model's input names
                  output_names = ['output'], # the model's output names
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable length axes
                                'output' : {0 : 'batch_size'}})
                                

I'll look in to optimising it with tools like OpenVINO if it does.

am also interested in onx export code of GDINO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants