Code

feat: add gemma2 variants#1835

Code

Merged

ebsmothers merged 12 commits intopytorch:mainpytorch/torchtune:mainfrom Optimox:feat/add-gemma-2Optimox/torchtune:feat/add-gemma-2Copy head branch name to clipboardNov 8, 2024

+2,456-9

Commits

Commits on Oct 15, 2024

• feat: add gemma2b variants

  Optimox

  committed
  9685e0b

  Copy full SHA for 9685e0b

Commits on Oct 18, 2024

• WIP: working gemma2 2b pipeline

  Optimox

  committed
  e87f878

  Copy full SHA for e87f878

Commits on Oct 22, 2024

• WIP: non working flex attention

  Optimox

  committed
  6f89920

  Copy full SHA for 6f89920

Commits on Oct 24, 2024

• update recipes for 9b and 27b

  Optimox

  committed
  0d53660

  Copy full SHA for 0d53660

Commits on Oct 26, 2024

• fix mlp and kv cache, disable flex attention

  Optimox

  committed
  6b50916

  Copy full SHA for 6b50916

Commits on Oct 27, 2024

• update configs to match parallel PR

  Optimox

  committed
  54a237c

  Copy full SHA for 54a237c

Commits on Oct 30, 2024

• remove flex attention

  Optimox

  committed
  2c216de

  Copy full SHA for 2c216de

Commits on Oct 31, 2024

• Update docs/source/api_ref_models.rst

  Show description for 5127db9
  

  joecummings

  and

  ebsmothers

  authored
  5127db9

  Copy full SHA for 5127db9

• Update docs/source/api_ref_models.rst

  joecummings

  authored
  599c828

  Copy full SHA for 599c828

Commits on Nov 4, 2024

• fix: update mask for causal sliding window attention

  Optimox

  committed
  7adb55a

  Copy full SHA for 7adb55a

• Merge branch 'feat/add-gemma-2' of github.com:Optimox/torchtune into feat/add-gemma-2

  Optimox

  committed
  9d61cd7

  Copy full SHA for 9d61cd7

Commits on Nov 8, 2024

• add error for block masks

  Optimox

  committed
  53eed40

  Copy full SHA for 53eed40