Detectron2 is slow on Jetson Nano, is there a way to make faster?

[ahmedshafiq12]

I've installed detectron2 on jetson nano 2GB version, and when I ran it, it took almost 3 min for inferencing the first image (I've calculated the time after loading the model) and took an average 2 seconds for inferencing one image even after resizing it to (460*460) which is very slow

my hypothesis:
The masking layer draws too many points, is there a way to reduce that?
The masking layer takes time, is there a way to remove it?
The Jetson nano uses 4Gb of Swap memory, does adding more memory help that?
There is a jetson inference library that uses .onnx files, would that do something?

or is there another way to make faster or something I've done caused this?

I'm using a virtual environment (Python 3.6.9)
detectron2 = 0.4
PyTorch = 1.8 and torchvision (v0.9.0) that matches the PyTorch installation.

Instructions To Reproduce the Issue:
Requirements

Python = 3.6.9 virtual environment.
gcc & g++ ≥ 5.4 are required.

PyTorch v1.8 installation :

wget https://nvidia.box.com/shared/static/p57jwntv436lfrd78inwl7iml6p13fzh.whl -O torch-1.8.0-cp36-cp36m-linux_aarch64.whl
sudo apt-get install python3-pip libopenblas-base libopenmpi-dev libomp-dev
pip3 install Cython
pip3 install numpy==1.19.4 torch-1.8.0-cp36-cp36m-linux_aarch64.whl

torchvision v0.9.0 installation :

sudo apt-get install libjpeg-dev zlib1g-dev libpython3-dev libopenblas-dev libavcodec-dev libavformat-dev libswscale-dev
git clone --branch v0.9.0 https://github.com/pytorch/vision torchvision
cd torchvision
export BUILD_VERSION=0.9.0
python -m pip install -e torchvision

detectron2 v0.4 installation :

wget https://github.com/facebookresearch/detectron2/archive/refs/tags/v0.4.tar.gz
tar -xzf v0.4.tar.gz
python -m pip install -e detectron2-0.4

[github-actions]

You've chosen to report an unexpected problem or bug. Unless you already know the root cause of it, please include details about it by filling the issue template.
The following information is missing: "Instructions To Reproduce the Issue and Full Logs";