Code

Sampler Throughput Optimization#192

Code

Merged

masahi merged 30 commits intooctoml:batch-servingoctoml/mlc-llm:batch-servingfrom sunggg:feature/2024-Feb/optimize-samplersunggg/mlc-llm:feature/2024-Feb/optimize-samplerCopy head branch name to clipboardFeb 8, 2024

+1,279-560

Commits

Commits on Feb 2, 2024

• wip

  sunggg

  committed
  d848017

  Copy full SHA for d848017

• cleanup and fix the fallback path

  sunggg

  committed
  4d53073

  Copy full SHA for 4d53073

• more cleanup

  sunggg

  committed
  ec1f8a4

  Copy full SHA for ec1f8a4

• more cleanup

  sunggg

  committed
  798bcee

  Copy full SHA for 798bcee

• add sampler testcase

  sunggg

  committed
  b70c1ac

  Copy full SHA for b70c1ac

• wip - resolve merge conflict

  sunggg

  committed
  ab0be50

  Copy full SHA for ab0be50

• Merge remote-tracking branch 'upstream/batch-serving' into feature/2024-Feb/optimize-sampler

  sunggg

  committed
  c419d84

  Copy full SHA for c419d84

• resolved merge conflict, move metadata preparation earlier

  sunggg

  committed
  469e817

  Copy full SHA for 469e817

• more cleanup

  sunggg

  committed
  a60f6f8

  Copy full SHA for a60f6f8

• wip:logprob

  sunggg

  committed
  b4d8129

  Copy full SHA for b4d8129

Commits on Feb 4, 2024

• logprob works for sync

  sunggg

  committed
  36968be

  Copy full SHA for 36968be

• remove unncessary output tokens

  sunggg

  committed
  efd02f3

  Copy full SHA for efd02f3

• multi-decode, fallback path

  sunggg

  committed
  0fb6c11

  Copy full SHA for 0fb6c11

• tested fallback

  sunggg

  committed
  c12d394

  Copy full SHA for c12d394

• wip

  sunggg

  committed
  7c09570

  Copy full SHA for 7c09570

Commits on Feb 5, 2024

• remove the custom multinomial since there was no much impact. Add sampling tests

  sunggg

  committed
  f24cd22

  Copy full SHA for f24cd22

• more sampler test cases

  sunggg

  committed
  d00fab3

  Copy full SHA for d00fab3

• wip:mypy

  sunggg

  committed
  e105f97

  Copy full SHA for e105f97

Commits on Feb 6, 2024

• fix mypy

  sunggg

  committed
  4fe1906

  Copy full SHA for 4fe1906

• fix

  sunggg

  committed
  af408c9

  Copy full SHA for af408c9

• expose model config to higher-level APIs

  sunggg

  committed
  6cb2ae3

  Copy full SHA for 6cb2ae3

• reflect feedback

  sunggg

  committed
  371fed7

  Copy full SHA for 371fed7

• fix mypy

  sunggg

  committed
  a49fd16

  Copy full SHA for a49fd16

Commits on Feb 7, 2024

• remove index shuffling in random/greedy

  sunggg

  committed
  e08842f

  Copy full SHA for e08842f

• wip

  sunggg

  committed
  ced3c12

  Copy full SHA for ced3c12

• better

  sunggg

  committed
  8fb9f33

  Copy full SHA for 8fb9f33

• better

  sunggg

  committed
  baca214

  Copy full SHA for baca214

• more comments

  sunggg

  committed
  8314be2

  Copy full SHA for 8314be2

• reflect feedback

  sunggg

  committed
  d304df0

  Copy full SHA for d304df0

Commits on Feb 8, 2024

• reflect comment

  sunggg

  committed
  5f5d1c0

  Copy full SHA for 5f5d1c0