Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GPU-accelerated token generation (new quantization format) #1412

Merged
merged 9 commits into from
May 13, 2023
Merged

GPU-accelerated token generation (new quantization format) #1412

merged 9 commits into from
May 13, 2023

Commits on May 12, 2023

  1. CUDA kernel for q4_0 dequant. + mat. vec. mult.

    @JohannesGaessler
    JohannesGaessler committed May 12, 2023
    Configuration menu
    Copy the full SHA
    637be12 View commit details
    Browse the repository at this point in the history

  2. Added q4_1 via template

    @JohannesGaessler
    JohannesGaessler committed May 12, 2023
    Configuration menu
    Copy the full SHA
    12fc292 View commit details
    Browse the repository at this point in the history

  3. Added missing __syncthreads();

    @JohannesGaessler
    JohannesGaessler committed May 12, 2023
    Configuration menu
    Copy the full SHA
    7dc2f57 View commit details
    Browse the repository at this point in the history

  4. --gpu_layers -> --gpu-layers

    @JohannesGaessler
    JohannesGaessler committed May 12, 2023
    Configuration menu
    Copy the full SHA
    f0af475 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2023

  1. Shorter dequantize_mul_mat_vec line

    @JohannesGaessler
    JohannesGaessler committed May 13, 2023
    Configuration menu
    Copy the full SHA
    0986c2f View commit details
    Browse the repository at this point in the history

  2. q5_0 dequantize_mul_mat kernel

    @JohannesGaessler
    JohannesGaessler committed May 13, 2023
    Configuration menu
    Copy the full SHA
    9da44fd View commit details
    Browse the repository at this point in the history

  3. More readable dequantize_mul_mat_vec logic

    @JohannesGaessler
    JohannesGaessler committed May 13, 2023
    Configuration menu
    Copy the full SHA
    5a0ecf7 View commit details
    Browse the repository at this point in the history

  4. dequantize_mul_mat_vec kernels for q5_1, q8_0, f16

    @JohannesGaessler
    JohannesGaessler committed May 13, 2023
    Configuration menu
    Copy the full SHA
    bb0993e View commit details
    Browse the repository at this point in the history

  5. llama : offload "output" tensor to GPU too + coding style fixes

    @ggerganov
    ggerganov committed May 13, 2023
    Configuration menu
    Copy the full SHA
    ad8a9e6 View commit details
    Browse the repository at this point in the history