Add Augmentation on Test Set

[DimaMirana]

Add Augmentation on Test Set❔

I want to train my model using the

!python train.py --weights yolov5x.pt --data '../data.yaml' --epochs 5 --cache --img 416 --evolve
to generate the optimal hyperparameter value for augmentation and learning rate etc. Later I want to use that hyperparameter on the same model for my custom dataset. How do I add the augmentation parameter on train.py. In your tutorial on TTA(test time augmentation) just adding --augmentation is enough. Will it be same for the training set also? Also can I see total number of data generated after augmentation?
!python train.py --batch 16 --weights yolov5x.pt --data '../data.yaml' --epochs 5 --cache --img 416 --hyp '/runs/train/evolve/hyp_evolved.yaml' will this be the command for adding augmentation on the training set?

[glenn-jocher]

Augmentation does not generate new data, it simply views the existing data in different, random ways every time an image is used. Therefore an image is never seen twice the same way during training.

Training augmentation settings are defined in a hyperparameter file. You can use hyperparameter evolution to optimize these values to your custom training requirements.

yolov5/data/hyp.scratch.yaml

Lines 22 to 33 in 9b92d3e

	hsv_h: 0.015 # image HSV-Hue augmentation (fraction)
	hsv_s: 0.7 # image HSV-Saturation augmentation (fraction)
	hsv_v: 0.4 # image HSV-Value augmentation (fraction)
	degrees: 0.0 # image rotation (+/- deg)
	translate: 0.1 # image translation (+/- fraction)
	scale: 0.5 # image scale (+/- gain)
	shear: 0.0 # image shear (+/- deg)
	perspective: 0.0 # image perspective (+/- fraction), range 0-0.001
	flipud: 0.0 # image flip up-down (probability)
	fliplr: 0.5 # image flip left-right (probability)
	mosaic: 1.0 # image mosaic (probability)
	mixup: 0.0 # image mixup (probability)

Test time augmentation is not related to training augmentation. TTA simply runs inference at multiple image sizes and left-right flips. See TTA and Evolution tutorials for additional details:

YOLOv5 Tutorials

• Train Custom Data  🚀 RECOMMENDED
• Weights & Biases Logging  🌟 NEW
• Multi-GPU Training
• PyTorch Hub  ⭐ NEW
• ONNX and TorchScript Export
• Test-Time Augmentation (TTA)
• Model Ensembling
• Model Pruning/Sparsity
• Hyperparameter Evolution
• Transfer Learning with Frozen Layers  ⭐ NEW
• TensorRT Deployment

[DimaMirana]

Is there any way to optimize only these values during evolution? And also after evolution !python train.py --batch 16 --weights yolov5x.pt --data '../data.yaml' --epochs 5 --cache --img 416 --hyp '/runs/train/evolve/hyp_evolved.yaml
This is the command to run augmentation on train data right?

[glenn-jocher]

@DimaMirana the meta dictionary in train.py controls hyperparameter constraints which you can use to freeze parameters.

yolov5/train.py

Lines 543 to 572 in 9b92d3e

	# Hyperparameter evolution metadata (mutation scale 0-1, lower_limit, upper_limit)
	meta = {'lr0': (1, 1e-5, 1e-1), # initial learning rate (SGD=1E-2, Adam=1E-3)
	'lrf': (1, 0.01, 1.0), # final OneCycleLR learning rate (lr0 * lrf)
	'momentum': (0.3, 0.6, 0.98), # SGD momentum/Adam beta1
	'weight_decay': (1, 0.0, 0.001), # optimizer weight decay
	'warmup_epochs': (1, 0.0, 5.0), # warmup epochs (fractions ok)
	'warmup_momentum': (1, 0.0, 0.95), # warmup initial momentum
	'warmup_bias_lr': (1, 0.0, 0.2), # warmup initial bias lr
	'box': (1, 0.02, 0.2), # box loss gain
	'cls': (1, 0.2, 4.0), # cls loss gain
	'cls_pw': (1, 0.5, 2.0), # cls BCELoss positive_weight
	'obj': (1, 0.2, 4.0), # obj loss gain (scale with pixels)
	'obj_pw': (1, 0.5, 2.0), # obj BCELoss positive_weight
	'iou_t': (0, 0.1, 0.7), # IoU training threshold
	'anchor_t': (1, 2.0, 8.0), # anchor-multiple threshold
	'anchors': (2, 2.0, 10.0), # anchors per output grid (0 to ignore)
	'fl_gamma': (0, 0.0, 2.0), # focal loss gamma (efficientDet default gamma=1.5)
	'hsv_h': (1, 0.0, 0.1), # image HSV-Hue augmentation (fraction)
	'hsv_s': (1, 0.0, 0.9), # image HSV-Saturation augmentation (fraction)
	'hsv_v': (1, 0.0, 0.9), # image HSV-Value augmentation (fraction)
	'degrees': (1, 0.0, 45.0), # image rotation (+/- deg)
	'translate': (1, 0.0, 0.9), # image translation (+/- fraction)
	'scale': (1, 0.0, 0.9), # image scale (+/- gain)
	'shear': (1, 0.0, 10.0), # image shear (+/- deg)
	'perspective': (0, 0.0, 0.001), # image perspective (+/- fraction), range 0-0.001
	'flipud': (1, 0.0, 1.0), # image flip up-down (probability)
	'fliplr': (0, 0.0, 1.0), # image flip left-right (probability)
	'mosaic': (1, 0.0, 1.0), # image mixup (probability)
	'mixup': (1, 0.0, 1.0)} # image mixup (probability)

The command to run augmentation during training is:
python train.py

The hyp file used is defined here:

yolov5/train.py

Line 458 in 9b92d3e

parser.add_argument('--hyp', type=str, default='data/hyp.scratch.yaml', help='hyperparameters path')

[DimaMirana]

I'm really sorry to disturb you but can you give me an example where you run augmentation during training with specific augmentation? Also if I just run
python train.py
how will I can give the data folder path batch size etc.?

[glenn-jocher]

@DimaMirana training applies all augmentation hyperparameters by default. The hyp file used is defined here (point to another or modify this file if you'd like):

yolov5/train.py

Line 458 in 9b92d3e

parser.add_argument('--hyp', type=str, default='data/hyp.scratch.yaml', help='hyperparameters path')

For a full list of training arguments and their defaults see train.py argparser:

yolov5/train.py

Lines 453 to 489 in ee16983

	if __name__ == '__main__':
	parser = argparse.ArgumentParser()
	parser.add_argument('--weights', type=str, default='yolov5s.pt', help='initial weights path')
	parser.add_argument('--cfg', type=str, default='', help='model.yaml path')
	parser.add_argument('--data', type=str, default='data/coco128.yaml', help='data.yaml path')
	parser.add_argument('--hyp', type=str, default='data/hyp.scratch.yaml', help='hyperparameters path')
	parser.add_argument('--epochs', type=int, default=300)
	parser.add_argument('--batch-size', type=int, default=16, help='total batch size for all GPUs')
	parser.add_argument('--img-size', nargs='+', type=int, default=[640, 640], help='[train, test] image sizes')
	parser.add_argument('--rect', action='store_true', help='rectangular training')
	parser.add_argument('--resume', nargs='?', const=True, default=False, help='resume most recent training')
	parser.add_argument('--nosave', action='store_true', help='only save final checkpoint')
	parser.add_argument('--notest', action='store_true', help='only test final epoch')
	parser.add_argument('--noautoanchor', action='store_true', help='disable autoanchor check')
	parser.add_argument('--evolve', action='store_true', help='evolve hyperparameters')
	parser.add_argument('--bucket', type=str, default='', help='gsutil bucket')
	parser.add_argument('--cache-images', action='store_true', help='cache images for faster training')
	parser.add_argument('--image-weights', action='store_true', help='use weighted image selection for training')
	parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
	parser.add_argument('--multi-scale', action='store_true', help='vary img-size +/- 50%%')
	parser.add_argument('--single-cls', action='store_true', help='train multi-class data as single-class')
	parser.add_argument('--adam', action='store_true', help='use torch.optim.Adam() optimizer')
	parser.add_argument('--sync-bn', action='store_true', help='use SyncBatchNorm, only available in DDP mode')
	parser.add_argument('--local_rank', type=int, default=-1, help='DDP parameter, do not modify')
	parser.add_argument('--workers', type=int, default=8, help='maximum number of dataloader workers')
	parser.add_argument('--project', default='runs/train', help='save to project/name')
	parser.add_argument('--entity', default=None, help='W&B entity')
	parser.add_argument('--name', default='exp', help='save to project/name')
	parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
	parser.add_argument('--quad', action='store_true', help='quad dataloader')
	parser.add_argument('--linear-lr', action='store_true', help='linear LR')
	parser.add_argument('--upload_dataset', action='store_true', help='Upload dataset as W&B artifact table')
	parser.add_argument('--bbox_interval', type=int, default=-1, help='Set bounding-box image logging interval for W&B')
	parser.add_argument('--save_period', type=int, default=-1, help='Log model after every "save_period" epoch')
	parser.add_argument('--artifact_alias', type=str, default="latest", help='version of dataset artifact to be used')
	opt = parser.parse_args()

or simply run python train.py --help

$ python train.py --help

usage: train.py [-h] [--weights WEIGHTS] [--cfg CFG] [--data DATA] [--hyp HYP] [--epochs EPOCHS] [--batch-size BATCH_SIZE] [--img-size IMG_SIZE [IMG_SIZE ...]] [--rect]
                [--resume [RESUME]] [--nosave] [--notest] [--noautoanchor] [--evolve] [--bucket BUCKET] [--cache-images] [--image-weights] [--device DEVICE] [--multi-scale]
                [--single-cls] [--adam] [--sync-bn] [--local_rank LOCAL_RANK] [--workers WORKERS] [--project PROJECT] [--entity ENTITY] [--name NAME] [--exist-ok] [--quad]
                [--linear-lr] [--upload_dataset] [--bbox_interval BBOX_INTERVAL] [--save_period SAVE_PERIOD] [--artifact_alias ARTIFACT_ALIAS]

optional arguments:
  -h, --help            show this help message and exit
  --weights WEIGHTS     initial weights path
  --cfg CFG             model.yaml path
  --data DATA           data.yaml path
  --hyp HYP             hyperparameters path
  --epochs EPOCHS
  --batch-size BATCH_SIZE
                        total batch size for all GPUs
  --img-size IMG_SIZE [IMG_SIZE ...]
                        [train, test] image sizes
  --rect                rectangular training
  --resume [RESUME]     resume most recent training
  --nosave              only save final checkpoint
  --notest              only test final epoch
  --noautoanchor        disable autoanchor check
  --evolve              evolve hyperparameters
  --bucket BUCKET       gsutil bucket
  --cache-images        cache images for faster training
  --image-weights       use weighted image selection for training
  --device DEVICE       cuda device, i.e. 0 or 0,1,2,3 or cpu
  --multi-scale         vary img-size +/- 50%
  --single-cls          train multi-class data as single-class
  --adam                use torch.optim.Adam() optimizer
  --sync-bn             use SyncBatchNorm, only available in DDP mode
  --local_rank LOCAL_RANK
                        DDP parameter, do not modify
  --workers WORKERS     maximum number of dataloader workers
  --project PROJECT     save to project/name
  --entity ENTITY       W&B entity
  --name NAME           save to project/name
  --exist-ok            existing project/name ok, do not increment
  --quad                quad dataloader
  --linear-lr           linear LR
  --upload_dataset      Upload dataset as W&B artifact table
  --bbox_interval BBOX_INTERVAL
                        Set bounding-box image logging interval for W&B
  --save_period SAVE_PERIOD
                        Log model after every "save_period" epoch
  --artifact_alias ARTIFACT_ALIAS
                        version of dataset artifact to be used

[DimaMirana]

So if I want to run training on my custom data .yaml for 5 epoch with my finding hyperparameter after training then this will be the command?
!python train.py --batch 16 --weights yolov5x.pt --data '../data.yaml' --epochs 5 --cache --img 416 --hyp '/runs/train/evolve/hyp_evolved.yaml'
Also can you tell me --epoch 5 in --evolve mode mean? Does it mean that this model run for 5 epoch and on each epoch it evolves the best model 300 times to get the optimal hyperparams?
!python train.py --weights yolov5x.pt --data '../data.yaml' --epochs 5 --cache --img 416 --evolve

[glenn-jocher]

@DimaMirana see Hyperparameter Evolution Tutorial for complete details on evolution. Evolution and training are seperate. Evolution will return a set of hyps that you can then point to to train, i.e. python train.py --hyp your_hyps.yaml.

YOLOv5 Tutorials

• Train Custom Data  🚀 RECOMMENDED
• Weights & Biases Logging  🌟 NEW
• Supervisely Ecosystem  🌟 NEW
• Multi-GPU Training
• PyTorch Hub  ⭐ NEW
• ONNX and TorchScript Export
• Test-Time Augmentation (TTA)
• Model Ensembling
• Model Pruning/Sparsity
• Hyperparameter Evolution
• Transfer Learning with Frozen Layers  ⭐ NEW
• TensorRT Deployment

[DimaMirana]

got it thanks. Also how to cite your project in papers? Where is the papaer?

[DimaMirana]

Also where can I see the overall Accuracy of the model of all classes. not the precision and recall.

[DimaMirana]

also any image about the yolov5 network architecture?

[glenn-jocher]

@DimaMirana please see https://github.com/ultralytics/yolov5#citation for citations. You can use netron or tensorboard to visualize architecture, and also search issues for custom visualizations.

[DimaMirana]

how to visualize architecture using tensorboard?

[glenn-jocher]

@DimaMirana in theory you should be able to uncomment L335 in train.py to visualize model with tensorboard, though we've had various various levels of success in practice.

yolov5/train.py

Lines 329 to 335 in 9ccfa85

	# Plot
	if plots and ni < 3:
	f = save_dir / f'train_batch{ni}.jpg' # filename
	Thread(target=plot_images, args=(imgs, targets, paths, f), daemon=True).start()
	# if tb_writer:
	# tb_writer.add_image(f, result, dataformats='HWC', global_step=epoch)
	# tb_writer.add_graph(model, imgs) # add model to tensorboard

[DimaMirana]

Thanks for the info. Also where can I found the f1 score of each class after finishing training? the current metrrics gives the following.
Class Images Labels P R mAP@.5 mAP@.5:.95:

[glenn-jocher]

@DimaMirana F1 curves are computed and displayed in F1.jpg in your results directory, though it is not printed to screen. VOC example with YOLOv5x6:
https://wandb.ai/glenn-jocher/VOC

[DimaMirana]

is there any way to get F1 score of each class?

[DimaMirana]

Also when using colab is there any way to save the weights and other necessary files so that i can resume it from previous training on the next day?

[glenn-jocher]

is there any way to get F1 score of each class?

It seems like the best way to get F1 at each class would be to add it to the printed metrics in test.py here. You should note though that each class also has a highest F1 at a different confidence.

yolov5/test.py

Lines 227 to 230 in 9ccfa85

	# Print results
	pf = '%20s' + '%12i' * 2 + '%12.3g' * 4 # print format
	print(pf % ('all', seen, nt.sum(), mp, mr, map50, map))

[DimaMirana]

Thanks. Also when using colab is there any way to save the weights and other necessary files so that i can resume it from previous training on the next day?

[glenn-jocher]

@DimaMirana yes there's two tricks. You can set your --project directory to a mounted Google Drive, or you can also try the new W&B integration which should save your checkpoints to W&B now also. See https://wandb.ai/cayush/yolov5-dsviz-demo/reports/Object-Detection-with-YOLO-and-Weights-Biases--Vmlldzo0NTgzMjk

[DimaMirana]

Many thanks.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.