Code

SOAP optimiser; int4 fixes for 4090#1006

Code

Merged

bghira merged 9 commits intomainbghira/SimpleTuner:mainfrom feature/soapy-shampoopsbghira/SimpleTuner:feature/soapy-shampoopsCopy head branch name to clipboardSep 29, 2024

+547-33

Commits

Commits on Sep 29, 2024

• add SOAP optimiser

  

  bghira

  committed
  5698453

  Copy full SHA for 5698453

• add closure to step

  

  bghira

  committed
  36a67ab

  Copy full SHA for 36a67ab

• use fp32 gradients for soap

  

  bghira

  committed
  5785ce2

  Copy full SHA for 5785ce2

• update int4 logic to work on all ada series devices and fix SOAP

  

  bghira

  committed
  9402b65

  Copy full SHA for 9402b65

• default precision levels

  

  bghira

  committed
  3b16dab

  Copy full SHA for 3b16dab

• improve logging for cmd parser

  

  bghira

  committed
  286bcd2

  Copy full SHA for 286bcd2

• add cuda memory size check for SOAP

  

  bghira

  committed
  5ae00a3

  Copy full SHA for 5ae00a3

• soap -> fp32 grads

  

  bghira

  committed
  597e3dd

  Copy full SHA for 597e3dd

• fixes for soap after resume

  

  bghira

  committed
  e534c45

  Copy full SHA for e534c45