Multithreading benchmark #54
-
|
Appreciate a lot BM25S which is impressive, congrats for the lib ! I'm currently writing a BM25 using SQL and a database engine and I wanted to benchmark the number of Q/S against BM25s. The system I'm writing is designed to be multi-threaded and is dedicated to batch search. Results reported of BM25s are on a single thread. Is it possible to perform retrieval on multiple threads with BM25s to build a fair comparison? I tried Joblib + Delayed but encounter a pickle error Thank you |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
There's a built-in multithreaded approach, just set However the default doesn't seem to scale well, especially beyond 2 threads, you can see results here for 4 threads: https://github.com/xhluca/bm25-benchmarks The new numba JIT backend that will be added on v0.2.0 should be better for multithreading since it lets numba (i.e. llvm) handle the parallel processing (via |
Beta Was this translation helpful? Give feedback.
There's a built-in multithreaded approach, just set
n_threads=nto the desired number of threads. It uses the built-in python multiprocessing library so you don't need joblib/delayed.However the default doesn't seem to scale well, especially beyond 2 threads, you can see results here for 4 threads: https://github.com/xhluca/bm25-benchmarks
The new numba JIT backend that will be added on v0.2.0 should be better for multithreading since it lets numba (i.e. llvm) handle the parallel processing (via
prange), rather than doing it via python's multiprocessing. However it might be bottlenecked by memory access since it seems that at 4 threads it's not 100% efficiency (more like 50-60%, roughly).