[QNN] Change default rounding to UPWARD.

[anijain2305]

This PR is to change the default rounding from TONEAREST to UPWARD. QNN ops are converted to a sequence of Relay operator while compilation. TONEAREST rounding leads to many more relay operators as shown here

https://github.com/dmlc/tvm/blob/11a3a777ec78ad81ce12a940d07521870f546ad1/src/relay/qnn/util.cc#L113-L124

This PR makes a case for the default rounding option to be UPWARD.

I ran all TFLite hosted Inception Quantized models on a Cascade Lake machine with these two rounding options. I am giving the comparison numbers that show that UPWARD rounding is overall better alternative.

---

Accuracy Comparison

Accuracy is same regardless of the rounding choice. Tested over 50k images with pre-processing.

Model	Upward	Tonearest
Inception V1	69.592/89.456	69.592/89.456
Inception V2	73.332/91.328	73.332/91.328
Inception V3	77.306/93.618	77.306/93.618
Inception V4	79.55/94.196	79.55/94.196

---

Latency Comparison

Because TONEAREST leads to many more operators, the observed latency is higher.

Model	Upward (ms)	Tonearest (ms)
Inception V1	2.13497	2.97504
Inception V2	8.96444	11.89356
Inception V3	6.11635	9.13942
Inception V4	12.24914	16.80568

---

Compilation Time Comparison

ToNearest has very high compilation time. Though not included, it blows up drastically for mobilenet V1.

Model	Upward (seconds)	Tonearest (seconds)
Inception V1	80	113
Inception V2	53	55
Inception V3	67	80
Inception V4	64	75

---

Memory Footprint

Though I don't have numbers for that yet. @shoubhik also pointed out that memory footprint for the params increased with TONEAREST. This is mostly because TONEAREST introduces some more constants in the network, and they are in int64.

---

Background

Background about QNN rounding choices
https://github.com/dmlc/tvm/blob/11a3a777ec78ad81ce12a940d07521870f546ad1/include/tvm/relay/qnn/attrs.h#L52-L61

The point is that TONEAREST and UPWARD differ at only negative mid values (like -1.5, -2.5 and so on). This difference is invisible in end-to-end accuracy.

---

What does other frameworks do?

Every toolchain has their own favorite rounding choice. There is no widely chosen in-production standard AFAIK.

• TFLite - Chooses ARM based rounding. It works back from ARM assembly instructions, and very close to TONEAREST, but not exactly TONEAREST.
• MKLDNN - RoundNearestEven as per MKLDNN doc. This is not TONEAREST rounding.

---

Note that this PR is only changing the default value, one can still choose any rounding option if one wants. This is for TVM user, who does not want to worry about different roundings.

@tqchen @jackwish @vinx13 @FrozenGene @u99127 @zhiics @yzhliu

[tmoreau89]

Thank you for the thorough evaluation @anijain2305 ! Interesting results in terms of accuracy/performance tradeoffs. Let's just check with @ZihengJiang whether this would affect the quantization nightly runs.

[zhenhuaw-me]

Impressive! Thanks for the solid data! It's very interesting that the rounding policy impacts performance so much.

[zhiics]

Thanks. This looks great!

[zhiics]

Thanks everyone. This is now merged.

[zhiics]

@tmoreau89 Sorry. I didn't see your comment above. Should I revert?

[anijain2305]

This is QNN stuff. So, it would not affect automatic quantization nightly runs. So, this should be fine.

[tmoreau89]

@zhiics no problem, no need to revert