Investigate/fix test_cancel_running_job

[kt474]

This test has been failing for a while now - https://github.com/Qiskit/qiskit-ibm-runtime/actions/runs/5863436233/job/15896900505

It appears that when attempting to cancel a running job, calling job.cancel() can return a 204 and raise no errors while the job is actually completed.

This snippet shows that the status of job is CANCELLED but retrieving the job and status again shows DONE, which is the correct status.

I'm guessing this is some sort of race condition where the job runs too quickly and if this is the case, it should be handled appropriately on the client side.

def test_cancel_job_running(self, service):
        """Test canceling a running job."""
        job = self._run_program(service, iterations=5)
        if not cancel_job_safe(job, self.log):
            return
        time.sleep(10)  # Wait a bit for DB to update.
        rjob = service.job(job.job_id())
        print("job==rjob is " + str(job == rjob))
        print("job.job_id() = " + str(job.job_id()))
        print("rjob.job_id() = " + str(rjob.job_id()))
        print("job.status() = " + str(job.status()))
        print("rjob.status()) = " + str(rjob.status()))
        self.assertEqual(job.status(), JobStatus.CANCELLED)

[kt474]

For more context, here in the cancel method we are manually setting the job status to CANCELLED on line 252 even though in some cases the job is actually completed.

qiskit-ibm-runtime/qiskit_ibm_runtime/runtime_job.py

Lines 238 to 252 in 3d8b9c9

	def cancel(self) -> None:
	"""Cancel the job.
	Raises:
	RuntimeInvalidStateError: If the job is in a state that cannot be cancelled.
	IBMRuntimeError: If unable to cancel job.
	"""
	try:
	self._api_client.job_cancel(self.job_id())
	except RequestsApiError as ex:
	if ex.status_code == 409:
	raise RuntimeInvalidStateError(f"Job cannot be cancelled: {ex}") from None
	raise IBMRuntimeError(f"Failed to cancel job: {ex}") from None
	self.cancel_result_streaming()
	self._status = JobStatus.CANCELLED

[robotAstray]

@kt474 I believe there might be an issue both in the cancel() function race condition and test_cancel_job_running.
It appears that the job status is not updated correctly, which could be causing the problem. The Job is completing before the CANCELLATION requests goes through.

I think the following might resolve the problem:

1. Modify test function: Check the server-side status after using the cancel() method and wait for the state to be updated before proceeding with any assertions in test_cancel_job_running case.
2. Modify cancel() method: make sure cancel() method waits until the server-side status is updated before marking the job as "CANCELLED" on the client side.
Let me know what you think and perhaps I can try to implement this.

[merav-aharoni]

I think the test can be fixed quite simply by replacing the lines:

if not cancel_job_safe(job, self.log):
        return
time.sleep(10)  # Wait a bit for DB to update.
rjob = service.job(job.job_id())

by the lines:

rjob = service.job(job.job_id())
if not cancel_job_safe(rjob, self.log):
        return
time.sleep(10)

However, I think there is a deeper issue here, and that is why job and rjob are not identical. This does not have to do with the cancel itself.

[kt474]

Closing this for now, will reopen if it becomes an issue again