too mayn warning

[hee100]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I'm training a model and I keep getting WARNING that weren't generated before. WARNING persists even after epoch goes over 400.
Can I just keep to learn?

Changes

Does not occur before changes are applied. ( WARNING: NMS time limit 10.0s exceeded )

1. Changed object detection from two(pics > A:1800, B:1000) to three(pics > A:1800, B1000, C:500 ).
   1-1. The newly created class C has a WARNING( WARNIGN: corrupt JPEG restored and saved ) image at 25 train val each.

2. fl_gamma: 0.0 --> flgamma: 1.5

3. degree: 0.0 --> 20

4. mixup:0.0 --> 0.3

error

  Epoch   gpu_mem       box       obj       cls    labels  img_size
    61/449     29.5G   0.03907  0.003123  0.002231       664       640: 100%|██████████| 11/11 [00:48<00:00,  4.44s/it]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95:   0%|    | 0/2 [00:00<?, ?it/s]
WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95:  50%|▌| 1/2 [00:13<00:13, 13.23
WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|█| 2/2 [00:25<00:00, 12.78
                 all        687        706      0.687      0.128      0.121     0.0646

     Epoch   gpu_mem       box       obj       cls    labels  img_size
    62/449     29.5G   0.03979  0.003183  0.002103       687       640: 100%|██████████| 11/11 [00:17<00:00,  1.62s/it]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95:   0%|    | 0/2 [00:00<?, ?it/s]
WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95:  50%|▌| 1/2 [00:11<00:11, 11.91
WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|█| 2/2 [00:24<00:00, 12.36
                 all        687        706      0.487      0.216      0.205      0.122

     Epoch   gpu_mem       box       obj       cls    labels  img_size
    63/449     29.5G   0.03688  0.003107  0.002027       626       640: 100%|██████████| 11/11 [00:42<00:00,  3.90s/it]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95:   0%|    | 0/2 [00:00<?, ?it/s]
WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95:  50%|▌| 1/2 [00:12<00:12, 12.34
WARNING: NMS time limit 10.0s exceeded

Additional

train code
python train.py --img 640 --batch 256 --epochs 450 --data data/custom2.yaml --cfg models/yolov5n.yaml --weights yolov5n.pt --workers 16 --name goral_color_nano_mixup0.3 --patience 300

NOW hyp.yaml
lr0: 0.01

lrf: 0.1

momentum: 0.937


weight_decay: 0.0005

warmup_epochs: 3.0

warmup_momentum: 0.8

warmup_bias_lr: 0.1

box: 0.05

cls: 0.5

cls_pw: 1.0

obj: 1.0

obj_pw: 1.0

iou_t: 0.2

anchor_t: 4.0

fl_gamma: 1.5

hsv_h: 0.015

hsv_s: 0.7

hsv_v: 0.4

degrees: 20.0

translate: 0.1

scale: 0.5

shear: 0.0

perspective: 0.0

flipud: 0.0

fliplr: 1.0

mosaic: 1.0

mixup: 0.3

copy_paste: 0.0


[glenn-jocher]

@HeeHee100 your selection of hyperparameters is causing your problems, specifically fl_gamma.

If you don't know what you are doing and you want to update these see the Hyperparameter Evolution tutorial.

YOLOv5 Tutorials

• Train Custom Data  🚀 RECOMMENDED
• Tips for Best Training Results  ☘️ RECOMMENDED
• Weights & Biases Logging  🌟 NEW
• Supervisely Ecosystem  🌟 NEW
• Multi-GPU Training
• PyTorch Hub  ⭐ NEW
• TFLite, ONNX, CoreML, TensorRT Export 🚀
• Test-Time Augmentation (TTA)
• Model Ensembling
• Model Pruning/Sparsity
• Hyperparameter Evolution
• Transfer Learning with Frozen Layers  ⭐ NEW
• TensorRT Deployment

Good luck 🍀 and let us know if you have any other questions!

[hee100]

Thank you for your reply. I think it will be very helpful.
I have a few more questions.

1. I read the link you sent me.
   python train.py --epochs 10 --data coco128.yaml --weights yolov5s.pt --cache
   What functions does the "--cache" command have? ( + Where can I find out the explanation of other command? I couldn't find out even if I looked for the tutorial. )

2. I understood that Evolution was to find the best "Fitness" through many generations. So, the more epochs (ex-1500), the higher the probability of getting a better fitness?

3. I have a question in an article part about the hyperparameter evolution tutorial.

Initialize Hyperparameters

YOLOv5 has about 30 hyperparameters used for various training settings. These are defined in yaml files in the /data directory. Better initial guesses will produce better final results, so it is important to initialize these values properly before evolving. If in doubt, simply use the default values, which are optimized for YOLOv5 COCO training from scratch.

in this article, Better initial guesses will produce better final results, so it is important to initialize these values properly before evolving There is a writing like this.
Does mean this me having to do something before evolution?

[glenn-jocher]

@HeeHee100 see #4049

--cache Usage

python train.py  # default, no caching
python train.py --cache  # RAM cache
python train.py --cache ram  # RAM cache
python train.py --cache disk  # Hard Drive cache

2. Evolution is applied to a base scenario, the base scenario can include any number of --epochs you want. Your hyps will evolve for highest mAP constrained to those --epochs.

3. No. All hyps are already optimized for general usage case of training COCO on 300 epochs. You can use these safely to start your own evolution.

[sarmientoj24]

Still getting this error with VisDrone dataset and my fl_gamma is 0.

[bhnvx]

me either fl_gamma is 0, but still same error.

[stvogel]

Just to jump into the discussion ... I get the warning (sometimes) during inference (not during training).
... ok seems to be from general.py function non_max_suppression and just to break the loop if too much time is spent on NMS.

[anujdutt9]

Hi @glenn-jocher I see this same warning, training yolov5s on COCO dataset, with default parameters. The warning keeps showing up for 11 epochs and finally the training fails after that.

WARNING: NMS time limit 10.0s exceeded

Training Code Version: YOLOv5 v6.1

[glenn-jocher]

@anujdutt9 probably caused by dataset errors or non-default training settings/hyps

👋 hi, thanks for letting us know about this possible problem with YOLOv5 🚀. We've created a few short guidelines below to help users provide what we need in order to start investigating a possible problem.

How to create a Minimal, Reproducible Example

When asking a question, people will be better able to provide help if you provide code that they can easily understand and use to reproduce the problem. This is referred to by community members as creating a minimum reproducible example. Your code that reproduces the problem should be:

• ✅ Minimal – Use as little code as possible to produce the problem
• ✅ Complete – Provide all parts someone else needs to reproduce the problem
• ✅ Reproducible – Test the code you're about to provide to make sure it reproduces the problem

For Ultralytics to provide assistance your code should also be:

• ✅ Current – Verify that your code is up-to-date with GitHub master, and if necessary git pull or git clone a new copy to ensure your problem has not already been solved in master.
• ✅ Unmodified – Your problem must be reproducible using official YOLOv5 code without changes. Ultralytics does not provide support for custom code ⚠️.

If you believe your problem meets all the above criteria, please close this issue and raise a new one using the 🐛 Bug Report template with a minimum reproducible example to help us better understand and diagnose your problem.

Thank you! 😃