errors in pin-in-place path in HCC unpinned copy engine

[jeffdaily]

Using latest develop-upstream branch and latest benchmarks master.
Running the tf_cnn_benchmarks.py code like so:

python tf_cnn_benchmarks.py --num_gpus=4 --batch_size=64 --model=resnet50 --variable_update=parameter_server --local_parameter_device=cpu

Eventually produces during warmup the following message

terminate called after throwing an instance of 'Kalmar::runtime_exception'
  what():  HCC unpinned copy engine error
Aborted (core dumped)

If you set --local_parameter_device=gpu instead, the problem doesn't manifest.

However, the problem happens again even with --local_parameter_device=gpu during distributed training. Running 1 worker and 1 server like so:

# worker
python tf_cnn_benchmarks.py --local_parameter_device=gpu --num_gpus=4 --batch_size=64 --model=resnet50 --variable_update=distributed_replicated --ps_hosts=prj47-rack-05:50000 --worker_hosts=prj47-rack-02:50001 --job_name=worker --task_index=0 --server_protocol=grpc
# ps
python tf_cnn_benchmarks.py --local_parameter_device=gpu --num_gpus=4 --batch_size=64 --model=resnet50 --variable_update=distributed_replicated --ps_hosts=prj47-rack-05:50000 --wo^Cer_hosts=prj47-rack-02:50001 --job_name=ps --task_index=0 --server_protocol=grpc

At least with the distributed training, my guess is that tensors are moving from GPU to CPU prior to being packed into protobufs and shipped via grpc. Not sure why this is also happening during warm-up except that I specified the parameter device to be CPU, forcing a device to host copy for storing the params.

misc system info

c++ (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609

lscpu

AMD EPYC 7551 32-Core Processor

uname -a
Linux prj47-rack-02 4.13.0-43-generic #48~16.04.1-Ubuntu SMP Thu May 17 12:56:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

LD_LIBRARY_PATH /home/jdaily/openmpi-3.1.0-install/lib
DYLD_LIBRARY_PATH is unset

rocm-clang-ocl/Ubuntu 16.04,now 0.3.0-c1b678e amd64 [installed,automatic]
rocm-dev/Ubuntu 16.04,now 1.8.151 amd64 [installed]
rocm-device-libs/Ubuntu 16.04,now 0.0.1 amd64 [installed]
rocm-dkms/Ubuntu 16.04,now 1.8.151 amd64 [installed]
rocm-libs/Ubuntu 16.04,now 1.8.151 amd64 [installed]
rocm-opencl/Ubuntu 16.04,now 1.2.0-2018053053 amd64 [installed]
rocm-opencl-dev/Ubuntu 16.04,now 1.2.0-2018053053 amd64 [installed]
rocm-profiler/Ubuntu 16.04,now 5.4.6797 amd64 [installed]
rocm-smi/Ubuntu 16.04,now 1.0.0-42-g0ae1c36 amd64 [installed,automatic]
rocm-utils/Ubuntu 16.04,now 1.8.151 amd64 [installed]
rocminfo/now 1.0.7 amd64 [installed,local]

[whchung]

/cc @scchan

Please help collect more detailed log wrt copy with HCC_DB

[jeffdaily]

Logging a new error, same inputs to benchmark.

Memory access fault by GPU node-6 (Agent handle: 0x17e9ab0) on address 0x531981e000. Reason: Page not present or supervisor privilege.
Aborted (core dumped)

Gathering more detail next with HCC_DB.

[jeffdaily]

Ran with HCC_DB=0x48A. Output attached.
debug.txt

[whchung]

48A is good for diagnosing faulty kernels not very good for copies. Please try check HCC header file and use other flags so more details of the copies are shown.

[whchung]

https://github.com/RadeonOpenCompute/hcc/blob/clang_tot_upgrade/include/hc_rt_debug.h

Maybe 0x68A?

[jeffdaily]

I ran 0x0302 before your suggestion of 0x68A. Both attached. The 0x0302 run took longer for the error to appear. 0x68A showed the error relatively early.
debug_0x68A.txt
debug_0x0302.txt

[whchung]

Need more log especially from Python layer to understand which command we are in.

Also, to try make the behavior more deterministic, could you also try run the benchmark appending more env vars HCC_SERIALIZE_KERNEL=3 HCC_SERIALIZE_COPY=3?

[jeffdaily]

HCC_DB=0x68A
HCC_SERIALIZE_KERNEL=3
HCC_SERIALIZE_COPY=3
TF_CPP_MIN_VLOG_LEVEL=1

verbose1_serialize.txt.gz
It was 75MB uncompressed so I gzipped it before uploading.

[jeffdaily]

Setting

HSA_ENABLE_SDMA=0
HSA_ENABLE_INTERRUPT=0
HSA_DISABLE_CACHE=1

did not avoid the exception.

[jeffdaily]

I suppose it is worth noting that our TF1.3 sources ran fine.

I added func FILE and LINE output to the THROW_ERROR macro in unpinned_copy_engine.cpp, build hcc from source, and found the exception is thrown at line 193.
https://github.com/RadeonOpenCompute/hcc/blob/39bf3750d60b2a395650acf9c280fc3a819ba5c6/lib/hsa/unpinned_copy_engine.cpp#L193

hsaErr = 4096. Just a generic 0x1000 HSA_STATUS_ERROR.

[jeffdaily]

I dug into hsa_amd_memory_lock to see what causes it to return HSA_STATUS_ERROR. It returns it in two different cases, so I changed the return code in the first case to differentiate it from the latter, then installed ROCR-Runtime from source and reran the benchmark code. It still returns HSA_STATUS_ERROR, which tells me https://github.com/RadeonOpenCompute/ROCR-Runtime/blob/bfc4b9e98cb5e48d9f96287371d518cb444968ba/src/core/runtime/amd_memory_region.cpp#L611 it's failing in RegisterMemory in the call to hsaKmtRegisterMemoryToNodes().

[whchung]

Let's step back a bit. By "our TF1.3 sources ran fine" does it mean on the same lower-level bits (KFD / ROCR / HCC / HIP ) TF 1.3 + older TF benchmarks works while TF 1.8 + tip of TF benchmarks not?

[jeffdaily]

Yes, on the same lower-level bits TF 1.3 + older TF benchmarks works while TF 1.8 + top of TF benchmarks does not.

[jeffdaily]

Got a new error, but this time trying 1 ps and 1 worker. The worker failed during warmup with

Memory access fault by GPU node-7 (Agent handle: 0x24c6ab0) on address 0x4001d03000. Reason: Page not present or supervisor privilege.
Aborted (core dumped)

[whchung]

According to @jeffdaily the issue only happens on a particular configuration of ROCm stack and not observable in others. Awaiting more info.