-
-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sort speed increase #762
Sort speed increase #762
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this really helps!
Non-title sorting was the one part of the search revamp where I kinda ran out of time and defaulted to using the old code, but this looks much leaner in resources.
This'll still do a redis call for each ID(which is much better than the previous code), but I wonder if it wouldn't be possible to squeeze a bit more speed out of this by bundling all the hget
calls in a single database call using multi/exec
... Certainly some food for thought for next time.
Got a few suggested comments for the code in there, please have a look at 'em and correct me if I'm wrong 🙏
Co-authored-by: Difegue <8237712+Difegue@users.noreply.github.com>
Co-authored-by: Difegue <8237712+Difegue@users.noreply.github.com>
All comments are correct. I'd like to add other improvements, but I couldn't spend a lot of time. How about replacing Redis module to Redis::Fast module for incremental improvement? |
Looks perfect, thank you! |
Congratulations @djacks6278, you just earned a badge! Here it is: https://holopin.io/claim/clele32av01220fkyq7lbaioe This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account. |
Congratulations @djacks6278, you just earned a holobyte! Here it is: https://holopin.io/holobyte/clele4xlx08190fkyt49v88kr This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account. |
The current sorting algorithm runs hget every time and compares these values,
which has the disadvantage of heavy load and slow speed.
In addition, there was a huge problem with the existing alignment method.
The problem is that the larger the library, the more exponential it requires.
The load is also severe because hget is executed every operation.
The current method also tries to sort this value against the previous value each time.
To solve this problem,
The key values of the @filtered array and the value of the key value corresponding to the sort key are converted to Hash.
The key sorted based on the value value of this hash has been returned to the @sorted array.
We can process more quickly than before.
Currently, more than 100,000 sortings are often unsuccessful, but if you change them this way, 500,000 sortings can also be successful.
Based on the same performance for 100,000 sortings, traditionally it takes more than 300 seconds and the operation fails, but the improved code takes about 13 seconds.