Fix randomly failing test test_enqueue_once_after_enqueue

[jeremystretch]

Proposed Changes

The test netbox.tests.test_jobs.EnqueueTest.test_enqueue_once_after_enqueue occasionally fails for an unknown reason (see this example). This needs to be investigated and resolved.

Justification

CI tests should always pass reliably.

[bctiemann]

The exception is being raised from here, in rq/job.py --

        if refresh:
            status = self.connection.hget(self.key, 'status')
            if not status:
                raise InvalidJobOperation(f"Failed to retrieve status for job: {self.id}")
            self._status = JobStatus(as_text(status))

self.connection at that point when running locally is a redis.client.Redis instance. What is the setup in CI? Does it have Redis available? Or does this connection need to be mocked?

It seems like whatever caching backend is present in CI is intermittently failing to return a result for the key of the job being enqueue_once'd:

        job1 = TestJobRunner.enqueue(instance, schedule_at=self.get_schedule_at())
        job2 = TestJobRunner.enqueue_once(instance, schedule_at=self.get_schedule_at(2))

            # If the job parameters haven't changed, don't schedule a new job and keep the current schedule. Otherwise,
            # delete the existing job and schedule a new job instead.
            if (schedule_at and job.scheduled == schedule_at) and (job.interval == interval):
                return job
            job.delete()

We are trying to delete the job (because it is being called with new parameters), but its key is not in the cache when the above rq code is called, so it raises the exception. Maybe this is because the first instance of the job has already completed by the time the second one is enqueued? But I would think the key would still be present and it would have a status of finished, rather than not being there at all.

Since I have Redis in my local environment, this always works properly, but I suspect the setup in CI is different.

[jsenecal]

HA! It was driving me mad yesterday, thanks for flagging this @jeremystretch