Add StopOnEnqueueCount control

If set, CLIntercept will terminate the application on achieved enqueue count.
Useful to debug applications with sporadic issues.

Signed-off-by: Igor Venevtsev <igor.venevtsev@intel.com>

docs/controls.md

@@ -711,6 +711,10 @@ The Intercept Layer for OpenCL Applications will wait for this many milliseconds
 
 If set to a nonzero value, the Intercept Layer for OpenCL Applications will cause all OpenCL APIs to return a successful error status.
 
+##### `StopOnEnqueueCount` (cl_uint)
+
+If set, CLIntercept will terminate the application on achieved enqueue count. Useful to debug sporadic issues in applications. In case issue is not visible in identified faulting kernel, not needed to wait application normal exit.
+
 ##### `NullContextCallback` (bool)
 
 If set to a nonzero value, the Intercept Layer for OpenCL Applications will force the context callback to be NULL.  With both context callback logging and NULL context callback set, the context callback will still be logged, but any application context callback will not be called.

intercept/src/controls.h

@@ -172,6 +172,7 @@ CLI_CONTROL( cl_uint,       AubCaptureStartWait,                    0,      "The
 CLI_CONTROL( cl_uint,       AubCaptureEndWait,                      0,      "The Intercept Layer for OpenCL Applications will wait for this many milliseconds before ending aub capture.")
 
 CLI_CONTROL_SEPARATOR( Execution Controls: )
+CLI_CONTROL(cl_uint,        StopOnEnqueueCount,                     UINT_MAX, "If set, CLIntercept will exit the application on achieved enqueue count")
 CLI_CONTROL( bool,          NoErrors,                               false, "If set to a nonzero value, the Intercept Layer for OpenCL Applications will cause all OpenCL APIs to return a successful error status." )
 CLI_CONTROL( bool,          NullContextCallback,                    false, "If set to a nonzero value, the Intercept Layer for OpenCL Applications will force the context callback to be NULL.  With both context callback logging and NULL context callback set, the context callback will still be logged, but any application context callback will not be called." )
 CLI_CONTROL( bool,          FinishAfterEnqueue,                     false, "If set to a nonzero value, the Intercept Layer for OpenCL Applications inserts a call to clFinish() after every enqueue.  The command queue that the command was just enqueued to is passed to clFinish().  This can be used to debug possible timing or resource management issues and will likely impact performance." )

intercept/src/intercept.h

@@ -1947,10 +1947,15 @@ inline uint64_t CLIntercept::getEnqueueCounter() const
 
 inline uint64_t CLIntercept::incrementEnqueueCounter()
 {
+    uint64_t enqueueCounter = m_EnqueueCounter.load();
+    if( enqueueCounter == m_Config.StopOnEnqueueCount ) {
+        log(">>> Stop enqueue counter " + std::to_string(enqueueCounter) + " reached - terminating application...\n");
+        exit(0);
+    }
+
     uint64_t reportInterval = m_Config.ReportInterval;
     if( reportInterval != 0 )
     {
-        uint64_t enqueueCounter = m_EnqueueCounter.load();
         if( enqueueCounter != 0 && enqueueCounter % reportInterval == 0 )
         {
             report();