[Relay][Pass] Support combine multiple dense op just into dense

[wrongtest-intellif]

Hi there, this PR is a minor modification to CombineParallelDense pass, refer to https://discuss.tvm.ai/t/yet-another-dense-op-combine-strategy/7126. The changes are:

• Add option "to_batch" (default to True) to control whether combine dense ops into batch_matmul or dense.

• Add implementation to combine dense ops into one large dense instead of batch_matmul, which take almost same logic with that of CombineParallelConv2D pass.

• Test cases for combine various shapes of elem-wise op followed.

• The new strategy can combine even ops of different output dims and may take better performance in circumstances where flat matmul operation is faster than equivalent batch_matmul operation.

[wrongtest-intellif]

cc @jroesch @icemelon9 @comaniac

[comaniac]

Thanks for the PR! I have two questions other than the comments:

1. How to trigger this change (i.e., to_batch=false) for an end user? It seems to me that you can configure it only by manually modifying the build_module or VM compiler and rebuilding TVM.

2. IIUC, the reason that ParallelDenseFlatCombiner derived from ParallelOpCombiner instead of ParallelOpBatchCombiner is it requires special processes to almost every function, so it seems no benefit to derive from ParallelOpBatchCombiner. Now the class hierarchy becomes:

   • ParallelDenseBatchCombiner <- ParallelOpBatchCombiner <- ParallelOpCombiner
   • ParallelDenseFlatCombiner <----------------------------------|

   Since I didn't find any other classes derived from ParallelOpBatchCombiner, should we simplify ParallelOpBatchCombiner class if we cannot make both ParallelDense*Combiner derive from it?

[wrongtest-intellif]

Thanks for the PR! I have two questions other than the comments:

1. How to trigger this change (i.e., to_batch=false) for an end user? It seems to me that you can configure it only by manually modifying the build_module or VM compiler and rebuilding TVM.

2. IIUC, the reason that ParallelDenseFlatCombiner derived from ParallelOpCombiner instead of ParallelOpBatchCombiner is it requires special processes to almost every function, so it seems no benefit to derive from ParallelOpBatchCombiner. Now the class hierarchy becomes:

   • ParallelDenseBatchCombiner <- ParallelOpBatchCombiner <- ParallelOpCombiner
   • ParallelDenseFlatCombiner <----------------------------------|

   Since I didn't find any other classes derived from ParallelOpBatchCombiner, should we simplify ParallelOpBatchCombiner class if we cannot make both ParallelDense*Combiner derive from it?

Thanks for your comments !

• In our practice we just manually call Python api mod = relay.transform.CombineParallelDense(3, False)(mod).
  Because this pass will change the shape, currently we have to manually call it (or relay.optimize(mod) for default optimization) before any auto-tuning step to consider the combined kernel shape and then build.

• How about improve to make ParallelOpBatchCombiner as an exposed optional pass (maybe in another PR)? It can be used like mod = CombineParallelOpToBatch("op_name", "batch_op_name", 3). This may serve the original idea of this class and users can combine various kinds of op flexibly. Of course, the use case may be rare in common network structures.

[comaniac]

LGTM. Thanks.
Would like to get comments from @jroesch @icemelon9 @jonso4 as well.

[MarisaKirisame]

IMO there should be a CombineParallelOp Pass that just call CombineParallelDense/Conv2d/etc in sequential. Can you add that?
Also, why CombineParallel instead of Batch? Ppl use static batching/dynamic batching to describe this process.

[tqchen]

@MarisaKirisame please manage the PR and merge after things everyone approves

[MarisaKirisame]

@tqchen got it.

[tqchen]

@MarisaKirisame please followup

[MarisaKirisame]

@wrongtest can you just make a pass that is nothing but a sequential of the 3 combine passes? that's all i think should be changed.

[wrongtest-intellif]

Sorry for too late, a wrapped function BatchingOps() is added.