Resnet50 - ResourceExhaustedError on R9 Nano?

[lukeiwanski]

Hello,

First of all, thanks for the effort!

I encountered problem while following instructions from https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/rocm_docs/tensorflow-install-basic.md and https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/rocm_docs/tensorflow-quickstart.md#tensorflows-tf_cnn_benchmarks

The TF version used is: http://repo.radeon.com/rocm/misc/tensorflow/tensorflow-1.3.0-cp27-cp27mu-manylinux1_x86_64.whl

The GPU used is R9 Nano:

Number of platforms                               1
  Platform Name                                   AMD Accelerated Parallel Processing
  Platform Vendor                                 Advanced Micro Devices, Inc.
  Platform Version                                OpenCL 2.1 AMD-APP.internal (2617.0)
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd cl_amd_object_metadata cl_amd_event_callback 
  Platform Host timer resolution                  1ns
  Platform Extensions function suffix             AMD

  Platform Name                                   AMD Accelerated Parallel Processing
Number of devices                                 1
  Device Name                                     gfx803
  Device Vendor                                   Advanced Micro Devices, Inc.
  Device Vendor ID                                0x1002
  Device Version                                  OpenCL 1.2 
  Driver Version                                  2617.0 (HSA1.1,LC)
  Device OpenCL C Version                         OpenCL C 2.0 
  Device Type                                     GPU
  Device Profile                                  FULL_PROFILE
  Device Board Name (AMD)                         Fiji [Radeon R9 FURY / NANO Series]
  Device Topology (AMD)                           PCI-E, 01:00.0
  Max compute units                               64

Following command causes ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[64,512,28,28]:
python tf_cnn_benchmarks.py --model=resnet50 --num_gpus=1
Gist for full log here: https://gist.github.com/lukeiwanski/f20596d0c7812b977a70d40e13f4a45d

Have you seen anything like that before?

Thanks,

[whchung]

@lukeiwanski since you are using TensorFlow 1.3 please kindly post the issue at:

https://github.com/ROCmSoftwarePlatform/tensorflow

Issues here are for cutting-edge development for TensorFlow 1.8/1.9+

[whchung]

Also could you try reduce batch size? Nano has 4GB VRAM and normally ROCm kernel driver has to reserve somewhere about 250MB for internal bookkeeping so that leaves us 3.75GB usable. I'd imagine resnet50 with batch size 64 be a bit challenging in this configuration.

[lukeiwanski]

@whchung Thanks will post there.

I am more than happy to use TF 1.8/1.9+ . Do you have a working pre-build .whl that I could use? Or alternatively could you point me to branch and / or instructions on how to build the cutting-edge version?

EDIT: The batch size reduction did the trick! Thanks!

[whchung]

We haven't published pre-built WHL packages yet. We shall publish them shortly after ROCm 1.8.2 release. Meanwhile, you can try build from source using the tip of develop-upstream which is checked by CI.

[lukeiwanski]

Sweet, thanks I will give it a go!