Could you release small/tiny/nano version of detector and descriptor?

[zhongqiu1245]

Hello, thank you for your amazing job!
I'm really interesting of your job and want to deploy DeDoDe on mobile devices(laptop, even CPU) for some self-driving works.
But I find it is too heavy for mobile device to run DeDoDeDescriptorB, DeDoDeDetectorL.
In my computer(RTX4060 mobile 8G), only 5.4 fps when inputs with 640*480 (tensorrt_fp16)
Could you release small/tiny/nano version of detector and descriptor?
Thank you in advance!

[Parskatt]

Sure, the easiest I guess would be using vgg11 and reducing layers further. Should be doable. Not sure how much performance will degrade.

[zhongqiu1245]

about 30fps in RTX4060 mobile 8G.

[Parskatt]

@zhongqiu1245 could you try out the small detector in the branch that references this issue?

Weights can be found here: https://github.com/Parskatt/DeDoDe/releases/tag/v2

[Parskatt]

It uses a VGG11 backbone and I reduced the number of layers at each scale from 8 -> 4 and cut the dimensionality in half. I think it should be about 3-4X faster than the _L detector. Could you verify?

[Parskatt]

Depending on your application it might also be possible to increase the framerate by batching, is this an option for you?

[zhongqiu1245]

@Parskatt
Sorry for reply so late.
I will verify this.
Thank you!

[zhongqiu1245]

@Parskatt
Thank you for your DetectorS!
The fps increases rapidly, but still lower than 30fps (15.9fps, DetectorS + DescriptorB, 640*480, tensorrt fp16).

So I reduce the shape of img to 320 * 240, then fps=25, almost there.
Could you release a small version of Descriptor? Like DescriptorS?
Maybe this can help DoDeDo breaks up the limitation of 30fps.
Thank you!

[Parskatt]

Sure, then I think we can also reduce descriptor size. Does 128 sound better? Is descriptor dimensinality a concern?

[zhongqiu1245]

Thank you for your reply !
128 sounds better.
Yes, dim is an important factor which can speed up/slow down the inference time of net.The dim is smaller, the speed is faster. However, if dim is too small, it will cause bad performance. I thought dim=64 before but I thought it maybe too small. 128 maybe better :)
Thank you for your generous!

[zhongqiu1245]

some details:
resolution: (480, 640)
preprocess: 19.606828689575195ms
detectorS: 16.09945297241211ms
descriptorB: 29.36267852783203ms
dualsoftmaxmatcher: 0.6873607635498047ms
postprocess: 0.14138221740722656ms
total: 65.89770317077637ms fps: 15.207663468720314
detectorS & descriptorB are trt_fp16

[Parskatt]

Okay, so seems like around 20fps is at least possible with current sizes.

Are you able to extract the times for the encoder/decoder parts of the network? Depending on what is taking most time might need to change enc architecture.

The final thing I guess would be to distill both networks into a single network.

[zhongqiu1245]

ok, I will try later.