WIP Multi-scale training + Mask handling on detr + Deformable Detr

[thibo73800]

This pull request will include

• Detr with masked attention (Done, to test / evaluate)

• Multi-scale training (Done, to test / evaluate)

• Deformable DETR evaluation

• Deformable DETR finetuning (might be with multi scale training but without masked deformable attention)

[thibo73800]

Here are the scores of DETR r50 with the eval script and scaling

   |  all  |  .50  |  .55  |  .60  |  .65  |  .70  |  .75  |  .80  |  .85  |  .90  |  .95  |

-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
box | 39.47 | 58.34 | 55.96 | 53.17 | 50.26 | 46.51 | 41.69 | 35.96 | 28.26 | 18.34 | 6.18 |
mask | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+

39.47 != 42 from the paper. I guess this is because the used script do not handle crowd gt

[thibo73800]

Deformable infernece on webcam
python webcam_inference.py deformable-detr

Detr inference on webcam
python webcam_inference.py detr

Detr finetuning with fixed size images:
python finetune_coco.py --data_dir ~/data/coco/ --batch_size 1 --target_batch 8 --image_size 376,672

Detr finetuning with multi-scale training (as the paper)
python finetune_coco.py --data_dir ~/data/coco/ --batch_size 1 --target_batch 8 --image_size
To build the custom cuda ops (for deformable DETR):

cd detr_tf/custom_ops/ms_deform_attn/
./build.sh

[thibo73800]

Next steps:

• eval script with deformable Detr
• Finetuning with deformable Detr

[PhanTask]

Hi @thibo73800 , thanks a lot for updating WIP version of Deformable DETR! Recently I was finetuning my own object detection dataset based on your DETR code and the result is great (saving models and loading models work great without issues too). I have two questions:

1. Do we need to change any part in the current finetune code in order to train Deformable DETR (e.g., get_losses, aggregate gradients)? Or do we only need to replace detr class and transformer class with deformable_detr and deformable_transformer?
2. Would there be any plan for you to release your implementation of TrackFormer? Would be excited to see a TensorFlow version of TrackFormer! It would be great if you decided to release your TrackFormer code in the future so that I can learn from it and take it as a reference. Thanks!

[thibo73800]

@PhanTask

1. Training/Finetuning involve some change in the training pipeline. Especially in the hungarian algorithm and in the loss computation. I'll work on that next week as well as evaluating the deformable model.

2. Currently there is no official plan to release our implementation of Trackformer. But If enough people look for such implementation on Tensorflow, we might considere at some point to releasing the model on this repo. However, if you're willing to wok on it, we can consider giving you access to the implementation so that you can implemet the code on this repository.

[PhanTask]

@thibo73800 Thank you! Yes, I would be very happy to contribute to a Tensorflow version of TrackFormer. Please consider giving me access to it at your convenience.

[wr0112358]

@thibo73800 Does the committed version allow training of deformable DETR?

[djramakrishna]

@thibo73800 thanks for your efforts ! is there any reason why the current version hasn't been merged?

[djramakrishna]

@thibo73800 If you're not actively working on this, I'd like to contribute to the implementation of Deformable DETR, but would be able to do so only on weekends, so my progress may not be fast, if at all that matters.

[Venkyyy88]

Could you kindly provide some insight, @thibo73800 , as to why the recent version has not yet been merged? Any specific reasons? Thank you for your efforts!