SSD performance optimization and benchmark script #10483
Conversation
|
There're huge improvements for SSD. |
example/ssd/benchmark_score.py
Outdated
| import time | ||
| #from dataset.iterator import DetRecordIter | ||
| #from config.config import cfg | ||
| #from evaluate.eval_metric import MApMetric, VOC07MApMetric |
| variances[2], variances[3]); | ||
|
|
||
| // [id, prob, xmin, ymin, xmax, ymax] | ||
| outputs[i * 6] = id - 1; |
There was a problem hiding this comment.
Do any tests exist currently for this op ?
|
Maybe we can also benchmark with pretrained ssd model? Since with my experience of benchmarking ssd with mkldnn on skylake, multi box_detection shouldn't take that long when using pretrained model, due to number of valid objects is much smaller than random initialized parameters. For resnet50 as body network, time of multi box_detection is about 2% of the total inference time and with vgg16_reduced it is 8%. |
|
I hold the same opinion with @kevinthesun , it is not precise to benchmark randomly intialized weights. |
|
So this would only help at the beginning of training? |
|
MultiBoxDetection is only used for inference. @zhreshold Is this accurate? |
|
I think this should still improve the inference speed, but we would like to see how this affecting the pre-trained models. Apparently not by such a huge margin. |
| mod.bind(for_training = False, | ||
| inputs_need_grad = False, | ||
| data_shapes = get_data_shapes(bs)) | ||
| mod.init_params(initializer=mx.init.Xavier(magnitude=2.)) |
There was a problem hiding this comment.
try load some pre-trained models to test the real perf
There was a problem hiding this comment.
Sure. Do you know where I can find pre-trained parameters for SSD? I didn't find them in data.mxnet.io.
|
@zhreshold @kevinthesun @piiswrong where we can get the pre-trained SSD data? |
|
@pengzhao-intel |
|
@kevinthesun Yes, pre-trained model can be used for inference benchmarking, while it still involving dataloader and dataset, the benchmark script appended with this PR is dummy data-based and might be more flexible, I guess it is a potential utility as well. What do y ou think? :) |
|
@juliusshufan Synthetic data is fine as along as pre-trained model is used. |
Conflicts: src/operator/contrib/multibox_detection.cc
|
LGTM for this PR, just wish the benchmarks could be updated for the pretrained models so we can have better ideas of the perf improvement. |
* add ssd benchmark * optimize MultiBoxDetectionForward * update default batch sizes for ssd benchmark * remove commented python code
* add ssd benchmark * optimize MultiBoxDetectionForward * update default batch sizes for ssd benchmark * remove commented python code
* add ssd benchmark * optimize MultiBoxDetectionForward * update default batch sizes for ssd benchmark * remove commented python code
Description
In this PR:
python benchmark_score.pySSD-VGG16 with 20 classes and shape (3, 300, 300), img/sec
python benchmark_score.py --network resnet50SSD-ResNet50 with 20 classes and shape (3, 300, 300), img/sec
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.
Changes
Comments