Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

4-bit Integer quantisation #27

Merged
merged 38 commits into from
Mar 29, 2023
Merged

4-bit Integer quantisation #27

merged 38 commits into from
Mar 29, 2023

Commits on Mar 29, 2023

  1. gq : attempt at n-bit quantization

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    8f45628 View commit details
    Browse the repository at this point in the history

  2. gq : add amax based method 3

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    b0a46fd View commit details
    Browse the repository at this point in the history

  3. gq : progress on method 2

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    da2de94 View commit details
    Browse the repository at this point in the history

  4. gq : method 4 (AVX2)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    aa5506c View commit details
    Browse the repository at this point in the history

  5. gq : method 4 (ARM)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    1fc11de View commit details
    Browse the repository at this point in the history

  6. gq : method 4 (AVX2 attempt) + method 5 (no min)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    dae323c View commit details
    Browse the repository at this point in the history

  7. gq : method 5 (ARM)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    349e917 View commit details
    Browse the repository at this point in the history

  8. gpt-2 : model conversion for Q4_0 quantization

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    ff4c653 View commit details
    Browse the repository at this point in the history

  9. ggml : Q4_0 quantization support (ggml_get_rows())

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    21514b7 View commit details
    Browse the repository at this point in the history

  10. gpt-2 : loading Q4_0 quantized model

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    ff54fda View commit details
    Browse the repository at this point in the history

  11. ggml : q4_0 quantization support

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    2219c11 View commit details
    Browse the repository at this point in the history

  12. ggml : q4_1 quantization support (seems to work for bigger models)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    5bdfce2 View commit details
    Browse the repository at this point in the history

  13. gpt-2 : add gpt-2-quantize tool for quantizing f32 GPT-2 models

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    b0cab89 View commit details
    Browse the repository at this point in the history

  14. ggml : 4-bit quantization works (only scalar for now)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    2f11888 View commit details
    Browse the repository at this point in the history

  15. gq : add method 6 (ARM)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    b82c27f View commit details
    Browse the repository at this point in the history

  16. ggml : vectorized mad q4_0 (ARM)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    2e75d8f View commit details
    Browse the repository at this point in the history

  17. ggml : vectorized quantize_row_q4_0 (ARM)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    b0c22a4 View commit details
    Browse the repository at this point in the history

  18. ggml : simplify mad q4_0 (ARM)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    3c757a4 View commit details
    Browse the repository at this point in the history

  19. ggml : minor indentations

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    e3ad879 View commit details
    Browse the repository at this point in the history

  20. gpt-j : support for 4-bit quantized model inference

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    8abcab4 View commit details
    Browse the repository at this point in the history

  21. ggml : GGML_ASSERT() instead of assert() where appropriate

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    c21972c View commit details
    Browse the repository at this point in the history

  22. gpt : avoid ggml_transpose on model tensors (new models!)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    4a56c5b View commit details
    Browse the repository at this point in the history

  23. gpt-2 : minor

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    904605c View commit details
    Browse the repository at this point in the history

  24. gpt-j : fix conversion for FP16 models (such as GPT-JT-6B)

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    441a38f View commit details
    Browse the repository at this point in the history

  25. ggml : add ggml_compute_forward_rope_f16()

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    5336828 View commit details
    Browse the repository at this point in the history

  26. gpt : fix memory usage computation

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    99af48e View commit details
    Browse the repository at this point in the history

  27. ggml : fix ggml_is_contiguous() to take into account blck size

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    6aae09e View commit details
    Browse the repository at this point in the history

  28. whisper : add whisper-qunatize tool

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    d0ac5eb View commit details
    Browse the repository at this point in the history

  29. whisper : add support for quantized models

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    37d427d View commit details
    Browse the repository at this point in the history

  30. whisper : mem usage based on model format type

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    e904a58 View commit details
    Browse the repository at this point in the history

  31. gpt : seems not worth to use FP16 for KV cache

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    63a8f62 View commit details
    Browse the repository at this point in the history

  32. gpt : support quantisation of f16 models files

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    519ce47 View commit details
    Browse the repository at this point in the history

  33. ggml : fixes for rpi4

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    9881c2b View commit details
    Browse the repository at this point in the history

  34. whisper : add Q4_1 model sizes

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    a85bc0f View commit details
    Browse the repository at this point in the history

  35. ggml : add WASM SIMD for Q4_0

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    c4f1403 View commit details
    Browse the repository at this point in the history

  36. utils : print quantization histograms

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    331a862 View commit details
    Browse the repository at this point in the history

  37. ggml : sync all changes from llama.cpp and whisper.cpp

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    154fcc3 View commit details
    Browse the repository at this point in the history

  38. ggml : finalize the Q4_1 quantization for ARM_NEON

    @ggerganov
    ggerganov committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    724c45d View commit details
    Browse the repository at this point in the history