Fail case when running caffe opencl branch with isaac

[listenlink]

Hi,
I am running caffe opencl branch (https://github.com/BVLC/caffe/tree/opencl) with isaac master branch on INTEL BROADWELL platform with below command:
./build/test/test.testbin --gtest_filter=NetTest/2.TestLossWeight, that will bring fail.
While comment out the line 94 to 141 on file https://github.com/ptillet/isaac/blob/master/lib/runtime/profiles.cpp, that will pass the test case.

Can you reproduce the fail case, seems there is some problem with the copy operation on predict_ logic?

[ptillet]

This all makes sense! The idea is that I ended up choosing top-5 to increase the prediction accuracy. But if you benchmark multiple kernels that modify one of their inputs, then you need to be careful to copy back the result. Can you confirm that setting N_TOP=1 resolves the issue?

[listenlink]

Hi, it still failed when setting N_TOP=1

[gongzg]

@ptillet could you reproduce this issue at your BDW machine?

[ptillet]

Yes, I will try this tomorrow. I suspect it should come from copying back the output's result. Perhaps a problem with queues. In the meantime, could you try N_TOP=1 and set modify_output = false ?

[listenlink]

Sorry, I still failed when setting N_TOP=1 and modify_output = false on my side

[ptillet]

Haha. That sounds pretty bad! I'll definitely take a look at it.

[ptillet]

Some updates: I've found a bunch of other bugs when trying to reproduce the issue with caffe. I've fixed some things, and I'm tuning ISAAC for intel's latest driver and double-precision on broadwell. I hope to have everything up and running by the end of the week-end.

[ptillet]

Could you try the latest master? It should not only fix bugs for clCaffe, but also add double-precision support and performance improvements with the latest Intel OpenCL 2.0 driver.

Caffe seems to work fine with ISAAC on my BROADWELL machine.

[gongzg]

@ptillet I tried the latest master with clcaffe. The test suite will not crash now, but still has some failures:

[ FAILED ] SGDSolverTest/2.TestSnapshot, where TypeParam = caffe::GPUDevice
[ FAILED ] SGDSolverTest/2.TestLeastSquaresUpdateWithEverything, where TypeParam = caffe::GPUDevice
[ FAILED ] SGDSolverTest/2.TestLeastSquaresUpdateWithWeightDecayMultiIter, where TypeParam = caffe::GPUDevice
[ FAILED ] SGDSolverTest/2.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice
[ FAILED ] AdaGradSolverTest/2.TestSnapshot, where TypeParam = caffe::GPUDevice
[ FAILED ] NesterovSolverTest/2.TestNesterovLeastSquaresUpdateWithEverything, where TypeParam = caffe::GPUDevice
[ FAILED ] AdaDeltaSolverTest/2.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice
[ FAILED ] AdaDeltaSolverTest/2.TestSnapshot, where TypeParam = caffe::GPUDevice
[ FAILED ] AdamSolverTest/2.TestSnapshot, where TypeParam = caffe::GPUDevice
[ FAILED ] AdamSolverTest/2.TestAdamLeastSquaresUpdateWithEverything, where TypeParam = caffe::GPUDevice
[ FAILED ] RMSPropSolverTest/2.TestRMSPropLeastSquaresUpdateWithEverything, where TypeParam = caffe::GPUDevice
[ FAILED ] RMSPropSolverTest/2.TestSnapshot, where TypeParam = caffe::GPUDevice
[ FAILED ] InnerProductLayerTest/2.TestGradientTranspose, where TypeParam = caffe::GPUDevice

And if I ran some of the test cases directly, such as :
test/test.testbin --gtest_filter=SGDSolverTest/2.TestLeastSquaresUpdateWithEverything
It may fail or success at random manner. But If I use the old version:

commit 6ac5e1f55b1cae59394758f823d5c58f57ca561d
Author: Philippe Tillet <ptillet@g.harvard.edu>
Date:   Fri Jan 1 05:44:28 2016 -0500

    Templates/Reduce1D: now properly loading 2D scalars, it always passes all of the float gpu test cases.

It always passes all GPU float type testing.

[ptillet]

I see, thanks. I'm also having issues with some tests passing when called individually but failing when running the entire test suite. I'll solve this ASAP.

[ptillet]

Sometimes, ISAAC's GEMM uses two kernels. The event returned by clBlasSgemm always corresponded to the first one, which led to some synchronization issues. Setting label=0 solved the problem because it forced the library to use only one kernel. Fixed in f226837.
Could you retry now?

[gongzg]

@ptillet Nice catch, and it works great with clcaffe now. Thanks for your quick fix!

[ptillet]

That's good to hear :)