Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cuda : improve text-generation and batched decoding performance #3776

Merged
merged 7 commits into from
Oct 27, 2023
Merged

cuda : improve text-generation and batched decoding performance #3776

merged 7 commits into from
Oct 27, 2023
+125 −19

Commits on Oct 25, 2023

  1. cuda : prints wip

    @ggerganov
    ggerganov committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    59d1232 View commit details
    Browse the repository at this point in the history

  2. cuda : new cublas gemm branch for multi-batch quantized src0

    @ggerganov
    ggerganov committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    52af782 View commit details
    Browse the repository at this point in the history

  3. cuda : add F32 sgemm branch

    @ggerganov
    ggerganov committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    16b60dd View commit details
    Browse the repository at this point in the history

  4. cuda : fine-tune >= VOLTA params + use MMQ only for small batches

    @ggerganov
    ggerganov committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    a3c2843 View commit details
    Browse the repository at this point in the history

  5. cuda : remove duplicated cuBLAS GEMM code

    @ggerganov
    ggerganov committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    4c6744b View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. cuda : add CUDA_USE_TENSOR_CORES and GGML_CUDA_FORCE_MMQ macros

    @ggerganov
    ggerganov committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    a4e15a3 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2023

  1. build : add compile option to force use of MMQ kernels

    @ggerganov
    ggerganov committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    49af767 View commit details
    Browse the repository at this point in the history