Implement supervised fine-tuning #31
Conversation
|
SFT overfits a small dataset.
To reproduce: python repepo/algorithms/sft.py --sft.batch_size 4 --wandb.track TrueReference: https://wandb.ai/ucl-dark/Rep-Eng/runs/zuuj7mvq?workspace=user-dtch1997 Notes:
|
|
Rebased onto |
|
|
||
| class EvalCallback: | ||
| def __init__(self, val_datasets: Dict[str, BaseDataset]): | ||
| self.metric_fns = Metrics() |
There was a problem hiding this comment.
Can we use the same Evaluator objects that we use for benchmarking? The val_dataset should respond to the same metrics as the test_dataset I would think. Unless the idea is that we can set a specific validator that should be used by SFT to pick the best performing result? Regardless, the Evaluator type already returns a float so would be suited to this
There was a problem hiding this comment.
Sounds good, I haven't looked closely at Evaluator class yet but will do so.
| torch.nn.utils.clip_grad.clip_grad_norm_(model.parameters(), 1.0) | ||
| optimizer.step() | ||
| scheduler.step() | ||
| print(f"epoch : {epoch} | step: {step} | loss: {loss}") |
There was a problem hiding this comment.
nit: would be better to use tqdm to get an updating progress bar rather than printing directly. It would also be good to add a way to disable this outputting to the screen, maybe with a param to run() called verbose: bool? We can figure that out later though potentially, as it's more polish than core functionality
There was a problem hiding this comment.
yeah let's do this later. I also want to figure out how to separate the display from the core functionality.
|
@chanind do you have any idea why |
|
Will think about how to integrate |
|
|
||
| new_pipeline = algorithm.run(pipeline, dataset=dataset) | ||
|
|
||
| # Skip testing outputs as they will be gibberish |
There was a problem hiding this comment.
You can try testing with larger_model instead of model - that's pythia 360m and seems to generate real answers, e.g. https://github.com/dtch1997/repepo/blob/main/tests/core/test_benchmark.py#L10. Alternatively, we could try overfitting on a few examples and just asserting that it does output the stuff we overfit on. For instance, we could overfit on a single wrong example, like "Paris is in" "Germany", just to verify that it is training.
There was a problem hiding this comment.
Okay. but not gonna do it immediately. Will create an issue in backlog
| return AutoTokenizer.from_pretrained("EleutherAI/pythia-70m") | ||
| return AutoTokenizer.from_pretrained( | ||
| "EleutherAI/pythia-70m", | ||
| model_max_length=128, # Required to avoid overflow error in SFT |
There was a problem hiding this comment.
👍 this is a good call, I feel like this has come up in other places too
repepo/algorithms/sft.pyWandbLoggerinstancerunloop, though I think maybe this should be derived from theBenchmarkandEvaluatorclassesmake_datasetfrom aDatasetSpecwhich allows using a custom split of the dataset.Notes
algorithm.runAPI to allow for logging and eval callbacks. Maybe we can just allowalgorithm.runto accept arbitrarykwargs.to_dictandfrom_dictmethods toExampleandCompletioninrepepo.core.types.