Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

more perfo with llamafile tinyblas #655

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

more perfo with llamafile tinyblas #655

wants to merge 1 commit into from
+176 −182

Conversation

Djip007
Copy link
Contributor

@Djip007 Djip007 commented Dec 11, 2024

some result:

cpu_info model_filename size test 0.8.17 t/s PR t/s
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp1 2.19 2.21
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp2 4.37 4.32
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp3 6.46 6.46
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp4 4.97 8.53
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp5 6.12 10.57
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp6 12.90 12.67
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp7 8.70 14.66
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp8 9.81 16.64
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp9 19.19 18.58
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp10 12.49 20.39
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp11 13.57 22.37
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp12 25.43 24.19
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp13 16.22 25.90
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp14 17.26 27.62
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp15 31.09 29.34
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp16 19.98 30.66
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp30 34.87 42.15
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp32 28.64 42.39
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp64 31.97 45.37
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp65 32.18 45.78
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp120 36.41 46.66
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp128 34.18 46.70
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp130 33.91 46.52
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp240 35.88 47.21
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp255 35.81 47.54
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp256 34.69 47.21
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp510 34.54 45.90
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp512 33.64 46.10
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp1024 33.36 45.23
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp1025 32.15 44.65
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp2048 32.76 44.40
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB tg128 2.19 2.20
cpu_info model_filename size test 0.8.17 t/s PR t/s
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp1 2.19 2.20
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp2 4.37 4.32
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp3 6.44 6.43
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp4 4.90 8.52
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp5 6.19 10.62
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp6 12.83 12.67
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp7 8.54 14.64
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp8 9.85 16.67
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp9 19.10 18.68
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp10 12.24 20.57
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp11 13.53 22.53
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp12 25.38 24.43
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp13 16.00 26.31
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp14 17.27 28.28
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp15 31.60 30.01
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp16 19.67 31.58
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp30 39.28 47.86
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp32 30.30 49.49
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp64 34.77 55.82
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp65 35.08 55.46
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp120 40.33 57.03
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp128 37.34 56.71
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp130 37.36 56.98
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp240 40.43 57.12
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp255 40.45 56.98
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp256 38.88 56.77
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp510 39.67 56.70
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp512 38.94 56.12
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp1024 38.42 55.44
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp1025 37.90 54.11
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp2048 37.56 53.74
AMD Ryzen 9 5950X 16-Core Processor (znver3) Mistral-Nemo-Instruct-2407.F16 22.81 GiB tg128 2.19 2.20
cpu_info model_filename size test 0.8.17 t/s PR t/s
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp1 2.50 2.52
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp2 4.98 4.89
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp3 4.02 7.29
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp4 9.91 9.89
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp5 12.33 12.31
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp6 7.91 14.78
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp7 9.26 17.12
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp8 7.30 19.44
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp9 11.78 21.63
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp10 24.02 24.03
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp11 14.41 26.08
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp12 15.67 28.40
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp13 11.70 29.89
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp14 18.10 31.80
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp15 34.31 33.99
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp16 20.57 35.55
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp30 50.37 51.51
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp32 36.26 51.72
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp64 50.78 56.70
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp65 54.97 56.60
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp120 61.28 63.76
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp128 47.93 63.91
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp130 61.80 63.94
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp240 64.80 66.97
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp255 64.09 67.12
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp256 60.79 66.98
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp510 59.59 64.95
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp512 56.96 64.96
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp1024 56.23 63.50
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp1025 55.21 61.80
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB pp2048 54.72 60.85
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.BF16 22.81 GiB tg128 2.50 2.52
cpu_info model_filename size test 0.8.17 t/s PR t/s
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp1 2.50 2.53
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp2 4.99 4.88
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp3 4.00 7.28
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp4 9.81 9.90
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp5 12.18 12.27
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp6 7.92 14.66
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp7 9.18 16.65
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp8 7.26 18.77
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp9 11.65 21.04
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp10 23.25 22.96
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp11 14.25 25.03
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp12 15.52 26.66
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp13 11.63 27.29
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp14 17.87 28.66
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp15 30.60 29.62
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp16 19.99 30.92
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp30 37.02 35.90
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp32 28.69 34.87
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp64 32.08 37.88
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp65 38.69 38.04
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp120 38.93 39.29
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp128 31.78 39.24
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp130 38.21 39.58
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp240 38.18 39.49
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp255 38.87 39.42
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp256 36.77 39.26
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp510 37.74 38.42
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp512 36.70 38.10
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp1024 36.44 37.94
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp1025 35.92 37.49
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB pp2048 35.78 36.97
AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (znver4) Mistral-Nemo-Instruct-2407.F16 22.81 GiB tg128 2.51 2.53

@Djip007 Djip007 marked this pull request as draft December 11, 2024 18:58
@Djip007
Copy link
Contributor Author

Djip007 commented Dec 11, 2024

For now I think there is a bug somewhere...

@Djip007
more perfo with llamafile tinyblas
b542e64 
- change dispache strategie (thanks:
ikawrakow/ik_llama.cpp#71 )
- more cache freindly
@Djip007
Copy link
Contributor Author

Djip007 commented Dec 11, 2024

OK I corrected this PR...
I'll try more, in some case (big/litle CPU) it can be slower... I' try to update the dispacher for dynamic one.
My hop is it may close the gap with ggml/tinyblas 'tg' to 🤞

Only have zen3/4 to bench so if you have some bench/diff it may by nice.

My bench use: mistral nemo:

./usr/bin/llamafile-bench -r 3 -m Mistral-Nemo-Instruct-2407.BF16.gguf -n 128 -p "1,1,1,2,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,30,32,64,65,120,128,130,240,255,256,510,512,1024,1025,2048"
./usr/bin/llamafile-bench -r 3 -m Mistral-Nemo-Instruct-2407.F16.gguf  -n 128 -p "1,1,1,2,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,30,32,64,65,120,128,130,240,255,256,510,512,1024,1025,2048"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
llamafile
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Djip007