Slow delete_pattern()

[jdbit]

Hi, I have a problem with delete_pattern() function.
This code with cache.delete_pattern() is very-very slow:

cache.delete_pattern("template.cache.some_cache*")

But when I rewrite the code, it works blazing fast:

keys = cache.keys("template.cache.some_cache*")
for o in keys:
    cache.delete(o)

Why the delete_pattern() is so slow? Is it a bug or I'm doing something wrong?

[WisdomPill]

Hey @jdbit sorry for replying late.

Well the issue is that cache.delete_pattern uses cache.scan_iter, which internally uses SCAN command and by default return 10 items at a time, so actually if cache.delete_pattern does not have itersize it deletes 10 items at a time.

On the other hand cache.keys, returns all the items and the you actually delete 1 item at a time, I would suggest to use cache.delete_many which will be even faster.

Finally, probably client.scan_iter return 10 items much slower than deleting 10 items one at a time.

P.S. I did not verify that, it is just my suspicion.

[WisdomPill]

Personally I would change cache.delete_pattern, to use keys and then delete every key using a redis pipeline, since I am not sure how many deletes could be done in one DEL command. Would you be interested in contributing and adding this feature under a feature flag? (like by adding a pipeline=False argument)

[jdbit]

Hey @WisdomPill,
thank you for your reply. I just tried delete_many() and it works faster, thank you for the suggestion! Sorry, I'm a beginner, so no idea how to work with the redis pipeline. I'll try to learn from the code in this repository, maybe I can figure out how it works. Thank you!

[rootart]

@WisdomPill I would be happy to try it and implement the suggested feature. There are some concerns when using keys command for databases with many keys as it blocks other clients and could lead to performance issues. Would it be fine to have it turned off by default as a safe option?

[WisdomPill]

@rootart you are right. There could be a safe way to use scan and a bigger iter_size but delete keys in one go and a more dangerous option using keys. Technically with a big enough iter_size it would get close to keys performance wise

[rootart]

Hey @WisdomPill
Will appreciate it if you could have a quick look at the pr I prepared. Many thanks!