CL_OUT_OF_RESOURCES when deploy resnet18 to ARM device

[JammyZhou]

Hi,

I tried deploying resnet18 model to my bubblegum96 96Boards (with PowerVR GPU) with script deploy_resnet_rpc.py. I ran into CL_OUT_OF_RESOURCES error below no matter what max_num_threads I set as mentioned in #530. Any suggestions?

`[20:20:15] src/runtime/opencl/opencl_device_api.cc:195: Initialize OpenCL platform 'Intel Gen OCL Driver'
[20:20:15] src/runtime/opencl/opencl_device_api.cc:220: opencl(0)='Intel(R) HD Graphics Haswell CRW GT3 Desktop' cl_device_id=0x7fa3ca8d8780
Traceback (most recent call last):
File "deploy_resnet_rpc.py", line 80, in
m.run()
File "/home/test/.local/lib/python2.7/site-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/contrib/graph_runtime.py", line 108, in run
self._run()
File "/home/test/.local/lib/python2.7/site-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/_ffi/_ctypes/function.py", line 183, in call
ctypes.byref(ret_val), ctypes.byref(ret_tcode)))
File "/home/test/.local/lib/python2.7/site-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/_ffi/base.py", line 62, in check_call
raise TVMError(py_str(_LIB.TVMGetLastError()))
tvm._ffi.base.TVMError: Except caught from RPC call: [12:20:50] src/runtime/module_util.cc:46: Check failed: ret == 0 (-1 vs. 0) [12:20:50] src/runtime/opencl/opencl_module.cc:223: Check failed: e == CL_SUCCESS OpenCL Error, code=-5: CL_OUT_OF_RESOURCES

Stack trace returned 10 entries:
[bt] (0) /home/bubblegum96/tvm/lib/libtvm_runtime.so(_ZN4dmlc10StackTraceB5cxx11Ev+0x104) [0x7fb0ada93c]
[bt] (1) /home/bubblegum96/tvm/lib/libtvm_runtime.so(_ZN4dmlc15LogMessageFatalD1Ev+0x30) [0x7fb0adb3f0]
[bt] (2) /home/bubblegum96/tvm/lib/libtvm_runtime.so(_ZNK3tvm7runtime17OpenCLWrappedFuncclENS0_7TVMArgsEPNS0_11TVMRetValueEPPv+0x808) [0x7fb0b1c8e0]
[bt] (3) /home/bubblegum96/tvm/lib/libtvm_runtime.so(ZNSt17_Function_handlerIFvN3tvm7runtime7TVMArgsEPNS1_11TVMRetValueEEZNS1_6detail17PackFuncVoidAddr_ILi8ENS1_17OpenCLWrappedFuncEEENS1_10PackedFuncET0_RKSt6vectorINS6_14ArgConvertCodeESaISC_EEEUlS2_S4_E_E9_M_invokeERKSt9_Any_dataOS2_OS4+0xa4) [0x7fb0b1ccbc]
[bt] (4) /home/bubblegum96/tvm/lib/libtvm_runtime.so(TVMFuncCall+0x5c) [0x7fb0aee424]
[bt] (5) /tmp/tmpHRYm1p/sym.o.so(+0x311c) [0x7fb03d611c]
[bt] (6) /tmp/tmpHRYm1p/sym.o.so(fuse_conv2d_broadcast_mul_broadcast_add_relu+0x4ac) [0x7fb03d5bcc]
[bt] (7) /home/bubblegum96/tvm/lib/libtvm_runtime.so(+0x4af80) [0x7fb0b04f80]
[bt] (8) /home/bubblegum96/tvm/lib/libtvm_runtime.so(+0x86e40) [0x7fb0b40e40]
[bt] (9) /home/bubblegum96/tvm/lib/libtvm_runtime.so(+0x856d4) [0x7fb0b3f6d4]

Stack trace returned 10 entries:
[bt] (0) /home/bubblegum96/tvm/lib/libtvm_runtime.so(_ZN4dmlc10StackTraceB5cxx11Ev+0x104) [0x7fb0ada93c]
[bt] (1) /home/bubblegum96/tvm/lib/libtvm_runtime.so(+0x4b3c8) [0x7fb0b053c8]
[bt] (2) /home/bubblegum96/tvm/lib/libtvm_runtime.so(+0x86e40) [0x7fb0b40e40]
[bt] (3) /home/bubblegum96/tvm/lib/libtvm_runtime.so(+0x856d4) [0x7fb0b3f6d4]
[bt] (4) /home/bubblegum96/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler16HandlePackedCallEv+0x2cc) [0x7fb0b25424]
[bt] (5) /home/bubblegum96/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler13SwitchToStateENS2_5StateE+0x2c4) [0x7fb0b2a3dc]
[bt] (6) /home/bubblegum96/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler15HandleNextEventEPNS0_11TVMRetValueEbPKNS0_10PackedFuncE+0x2d8) [0x7fb0b2b7d0]
[bt] (7) /home/bubblegum96/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession22HandleUntilReturnEventEPNS0_11TVMRetValueEbPKNS0_10PackedFuncE+0x1e0) [0x7fb0b25cb8]
[bt] (8) /home/bubblegum96/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession10ServerLoopEv+0x5c) [0x7fb0b2604c]
[bt] (9) /home/bubblegum96/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime13RPCServerLoopEi+0x84) [0x7fb0b2f17c]`

[tqchen]

There are some schedules which involve shared memory. In general, we need special schedules for mobile gpu. There is an ongoing effort on adding those

[JammyZhou]

@tqchen Thanks for your update. Is there any initial work I can try? Please let me know if there is something I can help :-)

[tqchen]

@merrymercy is the one who takes charge of this, and i believe we will have some release soonish(in a few weeks)

[eslambakr]

Hello @JammyZhou @tqchen @merrymercy
I am facing the same problem, Did you solve it?

Thanks in advance