Real time Object detection in agricultural applications #488

ManosMpampis · 2023-11-15T15:57:59Z

This PR aims to enhance the speed capabilities of Nanodet Object detection pipeline.
Add features in inference and training,
Add optimization options for TensorRT,
Ensure JIT script capabilities for embedded devices.
Add a new model nanodet-plus-fast with pretrained weights for object detection crops in RoboWeedMap dataset

…t_plus_fast model

fix optimization procedure and add dynamic and channel last implementations for faster inference

…to Real-Time-OD-In-AGRICULTURAL

tsampazk · 2023-11-20T18:56:29Z

From what i can tell, tests fail due to the new dependency pycuda introduced which fails to install during toolkit installation and subsequently results in an incomplete installation of the env.

At this stage i would suggest to investigate whether it is crucial to include the new dependencies as a first step to fix this. @passalis

ManosMpampis · 2023-11-20T19:09:37Z

The thing is, for real time applications in embedded devices it is essential to use TensorRT and must initialize pycude for TensorRT. Is it better to use a try: except: to informe the user to manually install pycuda and TensorRT?

tsampazk · 2023-11-21T08:15:48Z

The thing is, for real time applications in embedded devices it is essential to use TensorRT and must initialize pycude for TensorRT. Is it better to use a try: except: to informe the user to manually install pycuda and TensorRT?

Even better, if this is crucial only for embedded devices maybe we can only include it in the installation of the toolkit designed for the embedded devices. I suggest to investigate whether we can have a workaround where these options become available only when running on appropriate devices. @passalis any thoughts?

passalis · 2023-11-21T08:19:57Z

Can't we have a standalone script in projects that performs any conversions necessary?

ManosMpampis · 2023-11-21T08:59:18Z

I think I can manage that but, because it is an important implementation that can be used, I think it would be beneficial to be in the main Learner so it is easier to find from pip installations. The main reason that pycuda it is not installed it is because it can not find the nvcc compiler. Don't we install in during tool installation? I would write a script in projects for now but I think would be better to fix this installation problem.

passalis · 2023-11-21T09:23:40Z

We cannot assume that nvcc will be available, since we also support CPU-only installations. The simplest options are either to check if nvcc/pycuda and tensorrt are available in the learner before importing anything related to these or moving the optimization into a separate script. Both sounds ok to me, so we could go with the easiest to implement option.

passalis · 2023-11-21T09:26:53Z

If we go with the first one though, it is more complicated to handle the dependencies issues, which is the main issue right now as far as I understand.

fix nanodet_RepVGG_A0_416.yml and nanodet_g.yml, simplify vgg backbone transfer tensorRT dependencies into gpu installation small fixes into prints - docks

ManosMpampis · 2023-11-23T14:40:17Z

I have tested all tools in my machine with fresh installation as it is noted in: https://github.com/opendr-eu/opendr/blob/master/docs/reference/installation.md#installing-for-gpu

If it passes the Tests in CI I think we can continue with the review

tsampazk · 2023-11-24T13:50:06Z

I tested the gpu installation of this branch on a local machine and it works fine. Before moving on with the review, @ManosMpampis there are a lot of changes in the algorithm directory. Are these changes coming from a parent repo or similar, or are they your own modifications?

ManosMpampis · 2023-11-24T14:46:15Z

Some scripts are from the original repo of Nanodet that the tool was already taken from. autobach.py and torch_utils.py are modified from YoloV5 repo but most of the changes are mine. Do you think that I need to change the licence copyrights?

tsampazk · 2023-11-24T15:34:50Z

Some scripts are from the original repo of Nanodet that the tool was already taken from. autobach.py and torch_utils.py are modified from YoloV5 repo but most of the changes are mine. Do you think that I need to change the licence copyrights?

Ok then you can add what relevant information you think is valid here. I was mainly asking this because when we use code from other repositories as-is, we don't review it as part of OpenDR, but since you modified these files, i think we should.

ManosMpampis · 2023-11-26T21:34:46Z

Thank you for your comments @tsampazk and @passalis. I update the branch with your suggestions, tell me if you think anything else needs to change before we are ready to merge this PR.

tsampazk

Thank you @ManosMpampis for the new features, i've left some comments below.

docs/reference/object-detection-2d-nanodet.md

src/opendr/perception/object_detection_2d/nanodet/nanodet_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

update model heads to have unified initializations and update config_file_detail.md . delete unused parameter of yml files.

ManosMpampis · 2023-11-28T14:32:11Z

As I resolve this comment:
https://github.com/opendr-eu/opendr/pull/488/files/348d3cfe6064315ce9b9329820aaf92b25a21bfc#r1406131970 ,
I noticed that, config_file_detail.md in configs do not explain in detail the head parameters and even shows an unused parameter. I update the .md - delete the scales_per_octave parameter from models because it is unused. Last, I change the naming of 2 parameters in gfl_head.py and nanodet_plus_head.py to keep it consistent across all modules. Sorry for the inconvenient.

tsampazk

Thank you!

passalis · 2023-11-29T09:14:09Z

Thank you!

* add new augmentation strategies * add VGG backbone * easier code readability to add custom losses * better code readability * better code readability and small bug fixes * better and more stable training implementation and logging * add cache in dataset * add faster post processing in nanodet_plus_head.py and add new nanodet_plus_fast model * add TensorRT optimizations and fix embedded device inference, fix optimization procedure and add dynamic and channel last implementations for faster inference * add TensorRT optimizations and fix embedded device inference, fix optimization procedure and add dynamic and channel last implementations for faster inference * update docks and add warning ignores in test_nanodet * delete unused code, fix nanodet_RepVGG_A0_416.yml and nanodet_g.yml, simplify vgg backbone transfer tensorRT dependencies into gpu installation small fixes into prints - docks * fix a bug that still uses gpu nodes even self.device was set to "cpu" * update tool dock as noted in PR suggestions * Apply suggestions from code review Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * update dynamic input explanation accross optimization and inference. update model heads to have unified initializations and update config_file_detail.md . delete unused parameter of yml files. * fix TensorRT loading message and docs as suggested --------- Co-authored-by: ManosMpampis <gmp.manos@gmail.com> Co-authored-by: Nikolaos Passalis <passalis@users.noreply.github.com> Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

ManosMpampis added 4 commits November 15, 2023 17:33

add new augmentation strategies

a794f87

add VGG backbone

6090f1d

easier code readability to add custom losses

0759789

better code readability

65dfdbc

ManosMpampis changed the title ~~Real time od in agricultural~~ Real time Object detection in agricultural applications Nov 15, 2023

Merge branch 'develop' into Real-Time-OD-In-AGRICULTURAL

50e3ea4

passalis added the test sources Run style checks label Nov 20, 2023

ManosMpampis added 8 commits November 20, 2023 16:46

better code readability and small bug fixes

5bf1b5e

better and more stable training implementation and logging

147a1c4

add cache in dataset

cfa3475

add faster post processing in nanodet_plus_head.py and add new nanode…

663d42e

…t_plus_fast model

add TensorRT optimizations and fix embedded device inference,

92dcfd2

fix optimization procedure and add dynamic and channel last implementations for faster inference

add TensorRT optimizations and fix embedded device inference,

757db1c

fix optimization procedure and add dynamic and channel last implementations for faster inference

update docks and add warning ignores in test_nanodet

82965a0

Merge remote-tracking branch 'origin/Real-Time-OD-In-AGRICULTURAL' in…

0a1363e

…to Real-Time-OD-In-AGRICULTURAL

ManosMpampis marked this pull request as ready for review November 20, 2023 15:27

ManosMpampis requested review from passalis, omichel and stefaniapedrazzi as code owners November 20, 2023 15:27

tsampazk added the test tools Test the toolkit methods label Nov 20, 2023

delete unused code,

bce82aa

fix nanodet_RepVGG_A0_416.yml and nanodet_g.yml, simplify vgg backbone transfer tensorRT dependencies into gpu installation small fixes into prints - docks

ManosMpampis mentioned this pull request Nov 23, 2023

Nanodet jit and c fixes #490

Merged

ManosMpampis and others added 2 commits November 23, 2023 17:08

fix a bug that still uses gpu nodes even self.device was set to "cpu"

f657ec1

Merge branch 'develop' into Real-Time-OD-In-AGRICULTURAL

a3ace1a

Merge branch 'develop' into Real-Time-OD-In-AGRICULTURAL

d021dd8

update tool dock as noted in PR suggestions

348d3cf

tsampazk self-requested a review November 27, 2023 09:00

tsampazk requested changes Nov 28, 2023

View reviewed changes

ManosMpampis and others added 3 commits November 28, 2023 14:42

Apply suggestions from code review

da39ce0

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

update dynamic input explanation accross optimization and inference.

6e9d196

update model heads to have unified initializations and update config_file_detail.md . delete unused parameter of yml files.

fix TensorRT loading message and docs as suggested

b29f486

tsampazk approved these changes Nov 29, 2023

View reviewed changes

passalis approved these changes Nov 29, 2023

View reviewed changes

passalis merged commit 9429785 into develop Nov 29, 2023
48 checks passed

passalis deleted the Real-Time-OD-In-AGRICULTURAL branch November 29, 2023 09:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Real time Object detection in agricultural applications #488

Real time Object detection in agricultural applications #488

ManosMpampis commented Nov 15, 2023

tsampazk commented Nov 20, 2023

ManosMpampis commented Nov 20, 2023 •

edited

Loading

tsampazk commented Nov 21, 2023

passalis commented Nov 21, 2023 •

edited

Loading

ManosMpampis commented Nov 21, 2023

passalis commented Nov 21, 2023

passalis commented Nov 21, 2023

ManosMpampis commented Nov 23, 2023

tsampazk commented Nov 24, 2023

ManosMpampis commented Nov 24, 2023

tsampazk commented Nov 24, 2023

ManosMpampis commented Nov 26, 2023

tsampazk left a comment

ManosMpampis commented Nov 28, 2023

tsampazk left a comment

passalis commented Nov 29, 2023

Real time Object detection in agricultural applications #488

Real time Object detection in agricultural applications #488

Conversation

ManosMpampis commented Nov 15, 2023

tsampazk commented Nov 20, 2023

ManosMpampis commented Nov 20, 2023 • edited Loading

tsampazk commented Nov 21, 2023

passalis commented Nov 21, 2023 • edited Loading

ManosMpampis commented Nov 21, 2023

passalis commented Nov 21, 2023

passalis commented Nov 21, 2023

ManosMpampis commented Nov 23, 2023

tsampazk commented Nov 24, 2023

ManosMpampis commented Nov 24, 2023

tsampazk commented Nov 24, 2023

ManosMpampis commented Nov 26, 2023

tsampazk left a comment

Choose a reason for hiding this comment

ManosMpampis commented Nov 28, 2023

tsampazk left a comment

Choose a reason for hiding this comment

passalis commented Nov 29, 2023

ManosMpampis commented Nov 20, 2023 •

edited

Loading

passalis commented Nov 21, 2023 •

edited

Loading