[redis] use scan to iterate keys

[Rigo-m]

Describe the feature

Working with large redis db, you might encounter an error if you try to fetch too many keys altogether.
There should be a scan option available (or getKeys should be implemented via scan)

Additional information

• Would you be willing to help implement this feature?

[pi0]

Would you please share runnable reproduction?

[Rigo-m]

Hi pi0, using upstash as a redis provider
https://stackblitz.com/edit/nuxt-starter-ako5kk?file=server%2Fapi%2Ferror-redis.ts this simple script triggers the error when using getKeys.

Actually, as a second thought, I'm not sure whether the error is from upstash which is prohibiting reading that many keys altogether (since it is a blocking operation https://redis.io/commands/keys/) or whether it is a redis limitation.
Can I do anything else?

[pi0]

Hi @chronark do you know if it is a known limitation? And what would you suggest to scan too many keys?

[ps-20x]

For redis maybe there should be a match (scan) function that returns the keys by a matching pattern similar to the native scan function (https://redis.io/docs/latest/commands/scan/). The function could get the matching pattern (e.g. *name*), an optional a cursor (initial value 0) and an optional count (initial value is 10) similar to the native scan.

For us it is also not possible to work with the cache in vercel at the moment because we have too many entries (Error:

ERR KEYS command is disabled because total number of keys is too large, please use SCAN

[pi0]

Good idea, we might use match if base/maxDepth is provided.

I don't have time to immedtialy jump on it but feel free to open a PR 👍🏼 (it might be easiest if we introduce this behavior behind a driver flag to move forward faster)

[ps-20x]

Good idea, we might use match if base/maxDepth is provided.

I don't have time to immedtialy jump on it but feel free to open a PR 👍🏼 (it might be easiest if we introduce this behavior behind a driver flag to move forward faster)

Thanks for your response. Unfortunately we needed a solution for this asap. So we went around it and used our PHP Backend to connect to the redis and remove it from there. So I currently can't look into it. Maybe I'll find some time on the weekends.