[RFC][AUTOTVM] Counting FLOPs vs. OPs

[eqy]

AutoTVM has a handy floating point operation counter that has fit most use cases to this point, as almost all tuned workloads have been floating point. However, the recent introduction of the quantization pass means that we may be seeing non-floating point templates in the near-future. The current counter enforces that the data type at the bottom of every computation is some kind of a float, which breaks on tasks extracted from quantized graphs.

Should we ignore this for now (e.g., manually add "FLOP"s in this case), migrate to just counting "OPs", or something else like maintaining more than one counter?

CC @merrymercy @tqchen @ZihengJiang

[ZihengJiang]

I prefer ops, since we only need a metric for evaluating the computation overhead

[tqchen]

counting ops, or more specifically, MOPS(number of multiplications) is usually a good proxy for perf

[merrymercy]

Yes, we should move to OPs.
The reason I used FLOPs is to discard the integer multiplications in index expressions.
If we want to count integer OPs, should we discard them?

[tqchen]

seems we are converging to OPS, maybe we can summarize the decision and conclude by closing the RFC @eqy

[eqy]

Okay, the question is should we implement this change---@merrymercy has a point about whether we want to count index expressions. Is there an easy way to discard them?

[ajtulloch]

This came up in some of my int8 work, so I put together a pretty minimal PR at #2776 which I believe addresses this (modulo renaming compute_flops to something more general). To @eqy's point, it's pretty straightforward to ignore indexing expressions, and IMO ignoring them is the correct thing to do for essentially all the use-cases we have. If we were looking at extending this functionality, analytically computing the total amount of memory traffic (i.e. loads/stores) would be pretty useful, then we can essentially compute the roofline position of a given AutoTVM candidate which would be very helpful for stuff like e.g. early stopping.