DellEMC: No script is provided to run ssd-small

[psyhtest]

The README for SSD-Small using OpenVINO in the DellEMC submission, says:

Run the corresponding scripts under scripts/ folder

This folder, however, contains two files:

• 0_Warning.txt which says:

BLOCKED FILE ALERTA file has been blocked due to the 'Intel Email File Filtering Policy' rule.Context: 'ssdmobilenet_int8_offline.sh'Disallowed due to filenameTicket Number: '0ef4-5da0-0830-0001'
See your system administrator for further information. Copyright 1999-2011 McAfee, Inc.All Rights Reserved.http://www.mcafee.com

• ssdmobilenet_user.conf which mentions:

# Hardware: Intel(R) Xeon(R) 8280 CPU, 28 cores x 2

(which is neither Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz nor Intel(R) Xeon(R) Platinum 8276 CPU @ 2.20GHz that Dell EMC used in their submission.)

[psyhtest]

If I were to guess, this scripts/ folder should have been populated similarly to that in the corresponding Intel submission:

offline_ssdm.sh
offline_ssdm_accuracy.sh
server_ssdm.sh
server_ssdm_accuracy.sh
singlestream_ssdm.sh
singlestream_ssdm_accuracy.sh

[psyhtest]

Guessing further, SSD-Small in the Offline scenario and Performance mode would use offline_ssdm.sh:

Release_OMP/ov_mlperf --scenario Offline --mode Performance \
--mlperf_conf_filename "mlperf.conf" --user_conf_filename "scripts/ssdmobilenet_user.conf" \
--total_sample_count 5000 --data_path "/root/CK-TOOLS/dataset-coco-2017-val/" \
--dataset coco --model_path models/ssd-mobilenet/ssd-mobilenet_int8.xml \
--model_name ssd-mobilenet --nstreams 112 --nireq 112 --nthreads 112 \
--batch_size 1 --device cpu --nwarmup_iters 1000

[psyhtest]

To investigate a bit further, I've also decided to take a look at the measurements/ directory for the SSD-Small benchmark. To my surprise, it only contains one Intel entry, so only covers one Intel CPU (Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz) out of two and one scenario (Offline) out of two used in the submission. Moreover, user.conf is identical to the one Intel used in their submission (Intel(R) Xeon(R) Platinum 9282 CPU @ 2.60GHz gives it away immediately).

[psyhtest]

Curiously, neither the user.conf under measurements/ (intended for Intel(R) Xeon(R) Platinum 9282 CPU @ 2.60GHz)

*.Server.target_qps = 5260
*.Offline.target_qps = 10000

nor the ssdmobilenet_user.conf under code/ (intended for Intel(R) Xeon(R) 8280 CPU, 28 cores x 2):

*.Server.target_qps = 3000
*.Offline.target_qps = 4800

match the Dell systems well considering their respective Offline scores:

• Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz: 3744.24
• Intel(R) Xeon(R) Platinum 8276 CPU @ 2.20GHz: 4266.46

It's probably fine for the Offline mode, but could have led to suboptimal scores in the Server mode (not reported).

[psyhtest]

It's also peculiar that while the Offline score (inputs/second) is better (higher) for Intel(R) Xeon(R) Platinum 8276 CPU @ 2.20GHz (RCP: $8719) than for Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (RCP: $3075), the SingleStream score (milliseconds/sample) is better (lower) for the cheaper one: 1.54 vs. 1.69.

[psyhtest]

(Aside: It's a pity that no pricing is suggested for Intel(R) Xeon(R) Platinum 9282 CPU @ 2.60GHz.)

[psyhtest]

(Aside again: But for the best CPU for which pricing is available, Intel® Xeon® Platinum 8280L Processor @ 2.70GHz, RCP is $17906.)

[psyhtest]

(Yet another aside: In the preview category, Tencent used four Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz absent from the Intel site. The closest match I can find - 24 cores per CPU - is Intel® Xeon® Platinum 8260 Processor @ 2.40 GHz with RCP of $4702.)

[christ1ne]

The README for SSD-Small using OpenVINO in the DellEMC submission, says:

Run the corresponding scripts under scripts/ folder

This folder, however, contains two files:

• 0_Warning.txt which says:

BLOCKED FILE ALERTA file has been blocked due to the 'Intel Email File Filtering Policy' rule.Context: 'ssdmobilenet_int8_offline.sh'Disallowed due to filenameTicket Number: '0ef4-5da0-0830-0001'
See your system administrator for further information. Copyright 1999-2011 McAfee, Inc.All Rights Reserved.http://www.mcafee.com

• ssdmobilenet_user.conf which mentions:

# Hardware: Intel(R) Xeon(R) 8280 CPU, 28 cores x 2

(which is neither Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz nor Intel(R) Xeon(R) Platinum 8276 CPU @ 2.20GHz that Dell EMC used in their submission.)

Please ignore all comments regarding '8280', that was a mistake in the comment.
The actual CPU SKUs are reported correctly in the inference result table.

The Tencent SKU is a custom SKU so it is not reported on the Intel website.

[psyhtest]

Thanks @christ1ne. I had to look up SKU in Wikipedia (and donate £5 for its upkeep!)

The comment is wrong, but so is *.Offline.target_qps = 4800 for Dell EMC's CPUs?

[christ1ne]

My understanding is that you just need to set a number which is a reasonable for your HW. The final reported performance numbers will be from the logs.