New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Finetuning #48

Open

ktrapeznikov opened this issue Aug 31, 2022 · 1 comment

Assignees

Labels

Collaborator

ktrapeznikov commented Aug 31, 2022

No description provided.

ktrapeznikov added the experiment label

Collaborator Author

ktrapeznikov commented Sep 7, 2022 •

edited

Loading

baseline: trained on cam4 standard learning params (default batch size, lr 1e-3, 200 epochs)
SPCAM training from scratch (function of amount of data) for subsample = 1,2,4,8,16...
- lr 1e-3
- epochs 200 ?
- batch size ?
SPCAM fine-tuning baseline CAM4 (function of amount data) for subsample = 1,2,4,8,16 ....
- lr 5e-5
- epochs 200 ?
- batch size ?

how do we scale epochs or batch size as a function subsample?

Tasks:
(keep all the output vars)

Train CAM4,
Train SPCAM from scratch SPCAM for subsample = [1,2,4,8, ... 2^10]
Train SPCAM from CAM4 for subsample = [1,2,4,8, ... 2^10]

Setup to start:

scaling S, steps = N/B*T , if N/S -> B/S ?
try using same lr for fine-tuning and from scratch

Generate plot of skill vs N/S for both strategies

ktrapeznikov assigned blooyay

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment