llama : improve batched CPU perf with BLAS

[ggerganov]

We accidentally disabled BLAS when src0->type != F32

Also remove redundant check for full offload support

[slaren]

We should already limit the number of threads when using BLAS here:

llama.cpp/llama.cpp

Lines 5544 to 5551 in 0dab8cd

	// for big prompts, if BLAS is enabled, it is better to use only one thread
	// otherwise, the threads are spin-lock waiting for the BLAS calls and are degrading the performance
	// TODO: this is mostly important for Apple Silicon where CBLAS is still performing very well
	// we still need some threads to process all non-mul_mat ops, but not too much to avoid interfering
	// with the BLAS calls. need a better solution
	if (n_tokens >= 32 && ggml_cpu_has_blas() && !ggml_cpu_has_gpublas()) {
	n_threads = std::min(4, n_threads);
	}

[ggerganov]

But the way it was written, with n_threads == 8 and n_threads_batch == 1, it would use 1 thread for 1 < n_tokens < 32, while we want to use 8 threads.

[slaren]

Why would that happen, though? The purpose of n_threads_batch is using more threads when evaluating batches, since that's limited by compute and not by memory bandwidth. n_threads_batch should always be higher (or equal) than n_threads.

The threshold to start using n_threads_batch could be higher than 2, but the optimal value is likely to vary between different systems. But ultimately, the reason for n_threads_batch to exist is not to limit the number of threads when using BLAS, but rather to increase the number threads for computing batches when not using BLAS.

[ggerganov]

Yup, I got confused. Reverted that change

Edit: Github seems to be stuck

[FSSRepo]

Me too

https://www.githubstatus.com/incidents/66vhjmd266r9