Code

llama : per-layer KV cache#4309

Code

Merged

ggerganov merged 15 commits intomasterggerganov/llama.cpp:masterfrom gg/per-layer-kvggerganov/llama.cpp:gg/per-layer-kvCopy head branch name to clipboardDec 7, 2023

+747-287

Commits

Commits on Oct 3, 2023

• per-layer KV

  slaren

  committed
  e9bcf66

  Copy full SHA for e9bcf66

• remove unnecessary copies

  slaren

  committed
  55f2f2f

  Copy full SHA for 55f2f2f

Commits on Oct 6, 2023

• less code duplication, offload k and v separately

  slaren

  committed
  f4f9367

  Copy full SHA for f4f9367

Commits on Dec 3, 2023

• Merge branch 'master' into per-layer-kv

  ggerganov

  committed
  c294c78

  Copy full SHA for c294c78

• llama : offload KV cache per-layer

  ggerganov

  committed
  986b3da

  Copy full SHA for 986b3da

• llama : offload K shift tensors

  ggerganov

  committed
  f3dbfb9

  Copy full SHA for f3dbfb9

• llama : offload for rest of the model arches

  ggerganov

  committed
  3d3e6bd

  Copy full SHA for 3d3e6bd

• llama : enable offload debug temporarily

  ggerganov

  committed
  1fa91a4

  Copy full SHA for 1fa91a4

• llama : keep the KV related layers on the device

  ggerganov

  committed
  c44bc1e

  Copy full SHA for c44bc1e

• llama : remove mirrors, perform Device -> Host when partial offload

  ggerganov

  committed
  c80b8a2

  Copy full SHA for c80b8a2

• common : add command-line arg to disable KV cache offloading

  ggerganov

  committed
  e262947

  Copy full SHA for e262947

• llama : update session save/load

  ggerganov

  committed
  66aaac9

  Copy full SHA for 66aaac9

Commits on Dec 6, 2023

• llama : support quantum K cache (#4312)

  Show description for 1a1a1c3
  

  ggerganov

  and

  slaren

  authored
  1a1a1c3

  Copy full SHA for 1a1a1c3

Commits on Dec 7, 2023

• Merge branch 'master' into gg/per-layer-kv

  Show description for 680a99e
  ggerganov

  committed
  680a99e

  Copy full SHA for 680a99e

• readme : add API change notice

  ggerganov

  authored
  fc5f334

  Copy full SHA for fc5f334