No additional speed up after installing oneMKL

[azhuvath]

I got oneMKL installed in my CPU machine. Not sure if the llamafile is able to use it. Can we change the BLAS while executing? Not seeing any additional speed up after installing oneMKL.

[jart]

MKL is great, but llamafile gives you something that's oftentimes even better, which is tinyBLAS. Read about it here: https://justine.lol/matmul/

[jart]

If you think llamafile has room for improvement when it comes to BLAS on x86 then you can follow and participate in our latest efforts to improve it here: https://discord.com/channels/1089876418936180786/1258081988829642823/1275830898469765173

[azhuvath]

Thank you for looking into it. The flexibility to swap BLAS library will be a big advantage. With AI PC having CPU, iGPU, & NPU, targeting the right accelerator could influence both performance and power. Will read about the tinyBLAS.

[jart]

llamafile doesn't depend on any libraries and requests for us to support specific platforms has to be considered on a case by case basis, which is unlikely to happen since what we like to do as an open source project is focus on building generalized software on top of open architectures that's agnostic across vendors. The only BLAS library I know to exist that's smarter than what we're doing is cuBLAS which is so good and so popular that, despite being closed source, we gladly go out of our way to support it, since it serves the user's interests. But when it comes to CPU I'm an expert at BLAS and I'll eat my hat if you can find a library that knows something I don't.

[azhuvath]

What is the goal of this project? I thought to reach all the existing systems in the world without discrete accelerators from a presentation I saw about llamafile. LocalAI is the big movement that is happening today, moving a set of workloads from data center to the client and to the edge alleviating privacy concerns. It is clear that not everything can move to the client and edge.

Many existing system have integrated graphics which are not getting targeted. Added to that newer system like Intel Lunar Lake is reducing the CPU cores to minimum (4 P-Cores and 4 E-Cores) and adding more TOPS to iGPU and NPU. So this library by just targeting on CPU will be slower on newer laptops coming to the market. By targeting just CPU, older systems might work faster than newer ones because the newer ones are opting to add more TOPS to iGPU and NPU.

Open source is full of Yet Another Libraries (YAL) or Yet Another Framework (YAF) which are finding very difficult to optimize on a range of hardware's that are running in the ecosystem, probably a decades of hardware. In my observation the software ecosystem is failing to take advantage of the diversity present in the hardware where by many solutions are under utilizing the hardware. Only time will tell if llamafile will make a real difference in the plethora of software frameworks.