Optimize non-SIMD Q4 vector dot product

[sw]

This should help the poor souls running llama.cpp without a supported SIMD optimization.

First, the good stuff: per-token eval times in milliseconds:

With -march=native -mtune=native, but SIMD optimization code in ggml_vec_dot_q disabled:

	recent master (6e7801d)	this PR
Q4_0	2488	1320
Q4_1	2315	2023

Without -march=native -mtune=native, which causes the scalar code to be used without requiring modification:

	recent master (6e7801d)	this PR
Q4_0	2840	1330
Q4_1	2388	2083

Of course it would be interesting to see measurements from machines that actually do not have any SIMD circuits.
I would like to hear if this causes a regression in speed anywhere.
I can only really imagine this if floating point operations were faster than integer operations, because that's what we're trading here.
Beware of the recent change to the Makefile (c4f89d8), and possible differences to CMakeLists.txt, if you run your own tests.

The disadvantage is that this would once again cause a change in output due to floating point non-associativity / rounding.

I'll repost my snippet here, though admittedly this uses doubles:

$ echo 'main(){printf("%.16f\n%.16f\n",.7+.2+.1,.7+.1+.2);}' | gcc -x c - && ./a.out
0.9999999999999999
1.0000000000000000

We had changes like this before, and I understand that not having reproducible results makes some people unhappy.
I don't see us achieving this across the processor-specific optimizations without a severe regression in performance.

[sw]

I realize this merits checking the perplexity, but I get an ETA of 130 hours for the slowest case listed in the tables above. So roughly a month for a full run with all combinations :-(

[thement]

I realize this merits checking the perplexity, but I get an ETA of 130 hours for the slowest case listed in the tables above. So roughly a month for a full run with all combinations :-(

I'm not very versed in LLM and Neural Networks, but what is the exact point of checking the perplexity? Is that to check that the model hasn't regressed? Does it hold that the lower the perplexity the better the model output?

[sw]

I'm not very versed in LLM and Neural Networks, but what is the exact point of checking the perplexity? Is that to check that the model hasn't regressed? Does it hold that the lower the perplexity the better the model output?

That's basically it, though I wouldn't call myself an expert. If you look at some of the graphs in #406 that people have made, you'll see that the first few chunks have a large variation, so you'd ideally run the full set for it to give a good indication of quality.

[howard0su]

This change need more testing as scale smaller may lose some bits of m0/m1. I suggest you only change q4_0 which is current we are using. q4_0 change is safe which can be prove there are no bits losing.

[sw]

You are right that this is more questionable than the changes to Q4_0, and it also has less of a performance gain than Q4_0. I'll leave it for now so other people can try it and maybe comment on it. In the end we might revert that before merging.

[ggerganov]

Yes, let's keep only the Q4_0 change and merge

[ggerganov]

Yes, let's keep only the Q4_0 change and merge