Introduce Knowledge Distillation Base#432

Merged

ByronHsu merged 11 commits intomainfrom austin362667/feat/distill_base

Dec 9, 2024

+461-101

Commits on Dec 7, 2024

Add liger and naive distill base

austin362667
authored and
shivam15s
committed
Format

austin362667
authored and
shivam15s
committed
Refactor beta

austin362667
authored and
shivam15s
committed
Remove imports

austin362667
authored and
shivam15s
committed
Fix distill base chunk_size scaling

austin362667
authored and
shivam15s
committed
Fix chunk division

austin362667
authored and
shivam15s
committed
Remove chunk arg

austin362667
authored and
shivam15s
committed
Fix distillation_loss arg typo

austin362667
authored and
shivam15s
committed
use torch no grad and change normalization term
shivam15s
committed
rearrange fns for readability
shivam15s
committed
add no grad in tests
shivam15s
committed