How to fine a Flan T5 model on a single GPU for our dataset

[Rami-Ismael]

Dataset will be our baseline

1. We don't have a dataset yet with the entire physics dataset. Instead, we will use Scicen QA from 2022 as our benchmark. [1] to train, validate and test.

We using

1. Hugging Face Libraries
2. DeepSeed

Optimizer

1. The optimizer of choice will be determine computer resources
   1. Performance and Time
      1. Adam W will return to good performances. However, it needs * bytes time the number of parameters
      2. 8 Bit Adam W will return good performances. This will be slower. This method does dynamic quantization.

What GPU are you using

Current Parameters

Read this

• https://huggingface.co/docs/transformers/v4.26.1/en/main_classes/deepspeed#deployment-with-one-gpu
• [T5/MT5] resolve inf/nan under amp (mixed precision)  huggingface/transformers#10956
• https://www.philschmid.de/fine-tune-flan-t5-deepspeed
• getting nans with t5-large + fix huggingface/transformers#10830
• https://huggingface.co/blog/peft

References

[1]

@inproceedings{lu2022learn,
    title={Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering},
    author={Lu, Pan and Mishra, Swaroop and Xia, Tony and Qiu, Liang and Chang, Kai-Wei and Zhu, Song-Chun and Tafjord, Oyvind and Clark, Peter and Ashwin Kalyan},
    booktitle={The 36th Conference on Neural Information Processing Systems (NeurIPS)},
    year={2022}
}