Many mutex fails after #49000a30fd34ac21cabdc7d

[Saicheg]

Hey @pokonski !

We start seeing many mutex-failed issues after you commit #49000a30fd34ac21cabdc7d

Problem is that when you have 1000s of really fast-jobs they all start checking if final job should be
enqueued and conflicting too much.

I am thinking that you are actually wrapping wrong part with mutex, client.enqueue_job(workflow_id, out) feels like more right one

def enqueue_outgoing_jobs
  job.outgoing.each do |job_name|
    out = client.find_job(workflow_id, job_name)

     if out.ready_to_start?
       RedisMutex.with_lock("gush_enqueue_outgoing_jobs_#{workflow_id}-#{job_name}", sleep: 0.3, block: 2) do
          client.enqueue_job(workflow_id, out)
        end
      end
  end
end

What do you think? @pokonski

[pokonski]

Hey @Saicheg!

I'm not sure though if this will prevent two quick jobs from duplicating, because if they are checking without blocking they might have the outdated info and still en-queue the same child twice. What do you think?

[Saicheg]

@pokonski i think we actually should increase blocking timeout ( up to 5-10 seconds ) and instead of waiting for lock - we should fail all other jobs who are trying to enter that piece.

So even in case they will pass, only 1 job will enqueue final one and other will fail.

[Saicheg]

@pokonski do you have any chance to think through this?

[pokonski]

Yeah quietly failing makes sense, since the job will be enqueued anyway.

[schorsch]

I also got those mutex errors with 1200 jobs in the queue. I tried the proposed fix but when the last workers ran, the job to be run :after the batch got enqueued x-times, once for each running worker.

[treyperkins]

I was able to fix this by setting block to 0 and rescuing it. I found that since the jobs that couldn't get the mutex lock don't enqueue anything, the one that did acquire the lock ran and enqueued or skipped as appropriate.

However I did notice that in my case this only occurs on the first batch of jobs run on a Sidekiq server and only if they finish too closely to each other. Subsequent jobs on the same server do not seem to have this lock conflic issue.

[anirbanmu]

Was a resolution reached here on what the proper fix is?

[suonlight]

Can anyone help me to review my PR?

[pokonski]

PR was merged back then so I am closing it this issue :)