About memory bloat and the digest data model

[victorfgs]

Hey there 👋

I am currently monitoring usage in my production redis instance and some things came to my attention. For the past 4 months this server slowly but surely build up a little of a memory bloat and is currently holding approximately 75 keys matching the *uniquejobs* pattern even though there are no jobs running, scheduled or on the retry set.

Breaking down these 75k keys, there was a distribution in something like:

• ~13k keys matching the uniquejobs:*INFO* pattern
• ~30k keys matching the uniquejobs:*PRIMED* pattern
• ~69k keys matching the uniquejobs:*RUN* pattern

It was then that I realized that:

• There is some overlap between those patterns
• I have no idea how the data model of the gem works

This got me curious: how the gem organizes its data on redis? As far as I understood, there is a digest and this digest hold some info about the lock in question but I could not find documentation about how to use this info and I could not find about what any of these states represent (QUEUED and INFO are quite obvious) and why they were not deleted from my redis.

If these docs really do not exist I'd love to contribute to add them. This gem helps us everyday @QultureRocks and it will be a pleasure to help :)

Best,

[mhenrixon]

Hi @victorfgs,

So you answered the most obvious question by sharing the keys you are using. I know you are on v7+ version but which one specifically?

The next thing that would be helpful is to learn what types of locks you are using. There are some things about the until_expired locks that leaves some bloating but should soon be addressed by #721.

Explaining the lock mechanism is not easy because it depends on the lock itself. I'll get back to you next week on that because I don't have time to give the answer the attention to detail it deserves this week.

[victorfgs]

We're currently on version 7.1.23 with sidekiq on version 6.4.2 on ruby 2.7.5.

Most of our locks are until_and_while_executing with the strategies defined by on_conflict: { client: :log, server: :raise }. Some other jobs use the replace strategy on the client. I just realized that some jobs using this lock are implemented with no explicit strategies. This will soon be fixed. We do not use the until_expired lock.

The gem config we use is pretty much de default:

SidekiqUniqueJobs.configure do |config|
  config.enabled = !Rails.env.test?
  config.lock_info = ENV.fetch('SIDEKIQ_UNIQUE_JOBS_LOCK_INFO', 'false') == 'true'
end

[victorfgs]

Hi @mhenrixon

Any updates here?

[mhenrixon]

@victorfgs not yet, I have been on vacation and celebrated my birthday and the 5-year anniversary with my wife. Getting back to the computer now though so won't be much longer.

[mhenrixon]

So to expand a bit on the conversation. I needed a blocking way of waiting for an item. That's why queued and primed. When the queued entry is primed for locking meaning. I brpoplpush from queued to primed so I can finally consider the job locked.

Unfortunately, that call blocks all threads so that no redis work can be done. Since each sidekiq process has its thread but each worker belongs to the process thread. Concurrent::Ruby to the rescue :)

It isn't super convenient to model the data in redis. But:

1. I create a digest that we add as lock_digest to the job.
2. That digest is then used to a :QUEUED key (a list)
3. The digest is then moved from :QUEUED to :PRIMED to achieve waiting for a lock to be possible or fail.
4. A lock is created with the :LOCKED key (a redis hash). I can't remember the reason for this from the top of my head, but I remember it was a good reason 🤣.

When the lock has been achieved, the :QUEUED and :PRIMED keys are expired rather than deleted.

This is done to ensure that the waiting will work if someone is trying to queue or prime another job for the same digest. If I were to delete the keys straight out, a lot of weird stuff would happen, and things seemingly won't work.

Then as the job has been completed, the keys are cleared but... sometimes that doesn't happen. For example, I have been trying to fix that the PRIMED keys are not cleared up for the untile_expired locks.

Does that help at all @victorfgs?

[mhenrixon]

Oh yeah, I remember now. I add the current_time to the hash. I forgot about that, was going to use that for potentially stale lock detection.