[RELAY][OP] Dynamic conv2d batch size for cuda #6598
Conversation
| @@ -432,7 +441,8 @@ def nhwc_winograd_cuda( | |||
| name="output", | |||
| tag="conv2d_nhwc_winograd", | |||
| ) | |||
| cfg.add_flop(2 * N * CO * H * W * CI * KH * KW) | |||
| if isinstance(N, int): | |||
| cfg.add_flop(2 * N * CO * H * W * CI * KH * KW) | |||
There was a problem hiding this comment.
@kevinthesun @icemelon9 @comaniac is this okay to autotvm?
There was a problem hiding this comment.
It's okay in terms of the functionality, but the output message would be weird. Since the AutoTVM progress bar shows throughput instead of latency, users will always see 0 GFLOPS during the tuning process (https://github.com/apache/incubator-tvm/blob/master/python/tvm/autotvm/tuner/callback.py#L159).
Maybe we can still have the FLOPS with N=1 and pop a message saying we are tuning the kernel with N=1 but it can be used by the kernel with any batch size?
There was a problem hiding this comment.
yeah, I thought about 1 as well. But it actually maybe not 1
There was a problem hiding this comment.
I think it's fine since generally AutoTVM can't be used for dynamic shape op. User won't see any flops info when N is symbolic.
| @@ -432,7 +441,8 @@ def nhwc_winograd_cuda( | |||
| name="output", | |||
| tag="conv2d_nhwc_winograd", | |||
| ) | |||
| cfg.add_flop(2 * N * CO * H * W * CI * KH * KW) | |||
| if isinstance(N, int): | |||
| cfg.add_flop(2 * N * CO * H * W * CI * KH * KW) | |||
There was a problem hiding this comment.
It's okay in terms of the functionality, but the output message would be weird. Since the AutoTVM progress bar shows throughput instead of latency, users will always see 0 GFLOPS during the tuning process (https://github.com/apache/incubator-tvm/blob/master/python/tvm/autotvm/tuner/callback.py#L159).
Maybe we can still have the FLOPS with N=1 and pop a message saying we are tuning the kernel with N=1 but it can be used by the kernel with any batch size?
comaniac
changed the title
|
Thanks @zhiics @kevinthesun |
This PR enables dynamic conv2d for CUDA.
CC @kevinthesun @icemelon9 @mbrookhart @comaniac